wiki:ReleaseProcess

Version 60 (modified by exarkun, 4 years ago) (diff)

To update the front page release links, you need wiki editor permissions

This document is a draft.

jml wrote it, and wants to give it a lot of love very very soon. If you want to edit it before then, consider talking to jml

Overview

To release Twisted, we do a pre-release and then an actual release.

By the end of a Twisted release we'll have:

  • Tarballs for Twisted as a whole, and for each of its subprojects
  • Windows installers for the whole Twisted project
  • Updated documentation (API, howtos & book) on the twistedmatrix.com site
  • Updated download links on the twistedmatrix.com site
  • Announcement emails sent to major Python lists
  • Announcement post on http://labs.twistedmatrix.com
  • A tag in our Subversion repository marking the release

Prerequisites

To release Twisted, you will need:

  • Commit privileges to Twisted
  • Shell access to cube.twistedmatrix.com
  • Write permissions to /srv/www-data/twisted on cube (normally, www-data membership)
  • Write permissions to /srv/www-data/website/vhosts/twistedmatrix.com/documents
  • Permissions to edit WikiStart
  • Channel operator permissions for #twisted
  • Admin privileges for Twisted's PyPI packages
  • Membership of https://launchpad.net/~twisted-dev
  • Contributor status for http://labs.twistedmatrix.com

Version numbers

Twisted releases use a time-based numbering scheme. Releases versions like YY.MM.mm, where YY is the last two digits of the year of the release, MM is the number of the release in the year, and mm is the number of the patch release.

For example:

  • The first release of 2010 is 10.0.0
  • The second release of 2010 is 10.1.0
  • If 10.1.0 has some critical defects, then a patch release would be numbered 10.1.1
  • The first pre-release of 10.0.0 is 10.0.0pre1, the second is 10.0.0pre2

Every release of Twisted includes the whole project, the core and all subprojects. Each of these has the same number.

How to do a pre-release

  1. Check the milestone for the upcoming release
    1. Get rid of any non-critical bugs
    2. Get any critical bugs fixed
  2. Check for any regressions
  3. Read through the INSTALL and README files to make sure things like the supported Python versions are correct
    1. Check the required Python version.
    2. Check that the list matches the current set of buildbots.
    3. Any mistakes should be fixed in trunk before making the release branch
  4. Choose a version number. Remember to include the 'preN' suffix
  5. File a ticket
    1. Assign it to the upcoming release milestone
    2. Assign it to yourself
    3. Call it "Release <version-number>"
  6. Check buildbot to make sure all supported platforms are green (wait for pending builds if necessary).
  7. Make a branch (e.g. mkbranch Twisted release-4290)
  8. Run ./bin/admin/change-versions XX.YY.ZZpreN
  9. Commit the changes made by change-versions
  10. Run ./bin/admin/build-news .
  11. Commit the changes made by build-news
  12. Bump copyright dates in trunk/LICENSE, trunk/twisted/_version.py and trunk/README
  13. Make a temporary directory for the tarballs to live in (e.g. mkdir /tmp/twisted-release)
  14. Run ./bin/admin/build-tarballs . /tmp/twisted-release/
  15. Upload the tarballs to a public website (see #4353)
  16. Write the pre-release announcement
    1. Read through the NEWS file and summarize the interesting changes for the release
    2. Get someone else to look over the announcement before doing it
  17. Announce the pre-release on
    1. the twisted-python mailing list
    2. on IRC in the #twisted topic
    3. in a blog post, ideally labs.twistedmatrix.com

Pre-release announcment

The pre-release announcement should mention the important changes since the last release, and exhort readers to test this pre-release.

Here's what the 10.0.0pre1 release annoucement ought to have looked like:

Live from PyCon Atlanta, I'm pleased to herald the approaching
footsteps of the 10.0 release.

Tarballs for the first Twisted 10.0.0 pre-release are now available at:
 http://people.canonical.com/~jml/Twisted/

Highlights include:

 * Improved documentation, including "Twisted Web in 60 seconds"

 * Faster Perspective Broker applications

 * A new Windows installer that ships without zope.interface

 * Twisted no longer supports Python 2.3

 * Over one hundred closed tickets

For more information, see the NEWS file.

Please download the tarballs and test them as much as possible.

Thanks,
jml

How to release Twisted

  1. Remove the 'pre' from all files that mention it (see #4316)
    1. e.g. find . -type f -not -path '*.svn*' -not -name '*~' | xargs grep -n 'pre1' to find all files
  2. Add the quote of the release to the README
  3. Submit the ticket for review
  4. When the ticket is reviewed, merge the branch into trunk
  5. Tag the release
    1. e.g. svn cp svn+ssh://svn.twistedmatrix.com/svn/Twisted/branches/release-x-4290 svn+ssh://svn.twistedmatrix.com/svn/Twisted/tags/releases/twisted-10.0.0
  6. Create a new staging area for the release (e.g. mkdir /tmp/twisted-release)
  7. Run ./bin/admin/build-tarballs . /tmp/twisted-release/
  8. Build windows installers
    1. http://buildbot.twistedmatrix.com/builders/winxp32-py2.5-msi/
    2. http://buildbot.twistedmatrix.com/builders/winxp32-py2.6-msi/
    3. For "Branch" specify the tag, e.g. "tags/releases/twisted-10.0.0"
    4. Download the latest .msi files from from http://buildbot.twistedmatrix.com/builds/, save them in the staging directory
  9. Sign the tarballs and Windows installers
    1. e.g. md5sum Tw* | gpg -a --clearsign > twisted-10.0.0-md5sums.txt
    2. e.g. scp twisted-10.0.0-md5sums.txt cube.twistedmatrix.com:/srv/www-data/twisted/Releases/
    3. example of the result
  10. Upload to the official upload locations (see #2888)
    1. e.g. scp /tmp/twisted-release/Twisted-10.0.0.tar.bz2 cube:/srv/www-data/twisted/Releases/Twisted/10.0/
    2. For all subprojects, upload the tarballs to cube:/srv/www-data/twisted/Releases/<Project>/<First two parts of version number>/
    3. Windows installers go in the Twisted project
  11. Update the permissions on cube
    1. For all new directories, chmod 2775 $DIRECTORY
    2. For all new files, chmod 664 $FILE
    3. For all new files and directories, chgrp www-data $FILE_OR_DIR
    4. e.g. find /srv/www-data/twisted/Releases -name '*10.0.0*' | xargs chmod g+w
  12. Wait for the tarballs to be mirrored to tmrc
    • They are mirrored 15 minutes every hour
    • You can ask foom on #twisted to force the mirror
  13. Update the WikiStart and Downloads pages
    1. XXX: Describe this process exactly
  14. Update website documentation
    • See APIDocs
    • Get the dependencies
      • Twisted (export from tagged release branch)
      • Pydoctor
      • Nevow
      • Epytext
    • Run the unofficial build-docs script (see below)
    • Update the permissions
      • XXX: how?
    • cp the documents to /srv/.../documentation/10.0.0
      • XXX: full path please
    • rm -rf twisted/{vfs,web2} from the export
    • Run the unofficial build-api script (see below)
    • Move the API docs and update the permissions
      • cd /srv/www-data/website/vhosts/twistedmatrix.com/documents
      • mv ~exarkun/documentation/twisted-10.0.0/api/ ./10.0.0/
      • chown -R www-data.www-data 10.0.0/api/
    • Copy the book
    • Check the permissions
      • XXX: what permissions?
    • Test the generated docs
      • XXX: what to look for?
    • Change the "current" symlink
      • XXX: exact commands
    • Removing the staging area
  15. Write the release announcement (see below)
  16. Update ReleaseRevisions with the revision of this release
    • XXX: What revision exactly?
  17. Announce the release
    1. Send a text version of the announcement to: twisted-python@…, python-announce-list@…, python-list@…, twisted-web@…, twisted-jabber@…
    2. Update PyPI records
    3. Launchpad; https://launchpad.net/twisted/+announcements
      • Include a text version of the announcement and the new entries of the NEWS file
    4. http://labs.twistedmatrix.com
      • Post a web version of the announcements, with links instead of literal URLs
    5. Twitter, if you feel like it
    6. #twisted topic on IRC (you'll need ops)

Release announcement

The final release announcement should:

  • Mention the version number
  • Include links to the release tarballs & Windows installers
  • Summarize the significant changes in the release
  • Consider including the quote of the release
  • Thank the contributors to the release

Here's an example:

On behalf of Twisted Matrix Laboratories, I am honored to announce the
release of Twisted 10.0.

Highlights include:

 * Improved documentation, including "Twisted Web in 60 seconds"

 * Faster Perspective Broker applications

 * A new Windows installer that ships without zope.interface

 * Twisted no longer supports Python 2.3

 * Over one hundred closed tickets

For more information, see the NEWS file.

It's stable, backwards compatible, well tested and in every way an
improvement. Download it now from:

 http://tmrc.mit.edu/mirror/twisted/Twisted/10.0/Twisted-10.0.0.tar.bz2 or
 http://tmrc.mit.edu/mirror/twisted/Twisted/10.0/Twisted-10.0.0.win32-py2.5.msi

Many thanks to Jean-Paul Calderone and Chris Armstrong, whose work on
release automation tools and answers to numerous questions made this
possible. Thanks also to the supporters of the Twisted Software
Foundation and to the many contributors for this release.

jml

When things go wrong

If you discover a showstopper bug during the release process, you have three options.

  1. Abort the release, make a new point release (e.g. abort 10.0.0, make 10.0.1 after the bug is fixed)
  2. Abort the release, make a new pre-release (e.g. abort 10.0.0, make 10.0.0pre3 after the bug is fixed)
  3. Interrupt the release, fix the bug, then continue with it (e.g. release 10.0.0 with the bug fix)

If you choose the third option, then you should:

  1. Delete the tag for the release
  2. Recreate the tag from the release branch once the fix has been applied to that branch

Examples

Build API docs for website

See also APIDocs. See #4354.

from twisted.python._release import APIBuilder
from twisted.python.filepath import FilePath

APIBuilder().build(
    'Twisted', 'http://twistedmatrix.com/', 
    'http://twistedmatrix.com/trac/browser/tags/releases/twisted-10.0.0/',
    FilePath('twisted'),
    FilePath('api'))

Build howto documents for website

See #4355.

from twisted.python import _release as r
from twisted.python.filepath import FilePath
done = {}
for p in FilePath('doc').walk():
    if p.basename() == 'man':
        print 'processing manual pages:', p.path
        done[p] = True
        r.ManBuilder().build(p)
for p in FilePath('doc').walk():
    if p.basename().endswith('.xhtml'):
        if p.parent() not in done:
            print 'processing howto pages:', p.parent().path
            done[p.parent()] = True
            r.DocBuilder().build(
                '10.0.0', FilePath('doc/core/howto'), p.parent(),
                FilePath('doc/core/howto/template.tpl'),
                'http://twistedmatrix.com/documents/10.0.0/api/%s.html',
                False)

print 'building book:'
for p in done:
    print '     ', p.path
r.BookBuilder().build(FilePath('doc/core/howto'), done.keys(),
                      FilePath('doc/core/howto/book.tex'),
                      FilePath('book.pdf'))

Open questions

  • How do we manage the case where there are untested builds in trunk?
  • Should picking a release quote be part of the release or the pre-release?
  • What bugs should be considered release blockers?
    • Ultimately it's the RM's discretion
  • Should news fragments contain information about who made the changes?
  • Does this document miss anything relevant that ReleaseProcedure includes?
  • A thought for future releases: since we'd really like folks to download the prereleases and try them out, perhaps we should put the NEWS file on the web somewhere official, too, so they can see all the cool stuff they can try out?
    • XXX: jml doesn't know what this means any more

Bugs mentioned on this page

See also