Ticket #4543: version-control-release-process-4543-1.patch

File version-control-release-process-4543-1.patch, 20.4 KB (added by bramwelt, 3 years ago)

ReST Version of Release Process

  • docs/core/development/index.rst

    diff --git docs/core/development/index.rst docs/core/development/index.rst
    index 0485572..fb9152b 100644
    Development of Twisted 
    1313   naming
    1414   philosophy
    1515   security
     16   release
    1617   policy/index
    1718
    1819
  • new file docs/core/development/release.rst

    diff --git docs/core/development/release.rst docs/core/development/release.rst
    new file mode 100644
    index 0000000..36d5caf
    - +  
     1.. _release:
     2
     3Release Process
     4===============
     5
     6This document describes the Twisted release process. Although it is
     7still incomplete, every effort has been made to ensure that it is
     8accurate and up-to-date. There are plans to eventually move this
     9document into the source tree (`#4543`_).
     10
     11This process has only been tested on Linux, so we recommend that you do
     12the release on Linux.
     13
     14.. important:: Thinking about editing this document? DON'T. The only time
     15   changes to this document are allowed is during a release. The only
     16   changes that may be made are changes that are tested by the release
     17   engineer and found to actually work.
     18
     19If you want to make other changes to the release process, follow the
     20normal Twisted development process (contribute release automation
     21software that has documentation and unit tests demonstrating that it
     22works).
     23
     24.. _#4543: https://twistedmatrix.com/trac/ticket/4543
     25
     26
     27Outcomes
     28--------
     29
     30By the end of a Twisted release we'll have:
     31
     32* Tarballs for Twisted as a whole, and for each of its sub-projects
     33
     34* Windows installers for the whole Twisted project
     35
     36* Updated documentation (API, howtos & book) on the twistedmatrix.com site
     37
     38* Updated download links on the twistedmatrix.com site
     39
     40* Announcement emails sent to major Python lists
     41
     42* Announcement post on http://labs.twistedmatrix.com
     43
     44* A tag in our Subversion repository marking the release
     45
     46
     47Prerequisites
     48-------------
     49
     50To release Twisted, you will need:
     51
     52* Commit privileges to Twisted
     53
     54* Access to dornkirk.twistedmatrix.com as t-web
     55
     56* Permissions to edit the Downloads wiki page
     57
     58* Channel operator permissions for #twisted
     59
     60* Admin privileges for Twisted's PyPI packages
     61
     62* Membership of https://launchpad.net/~twisted-dev
     63
     64* Contributor status for http://labs.twistedmatrix.com
     65
     66
     67Version Numbers
     68---------------
     69
     70Twisted releases use a time-based numbering scheme. Releases versions
     71like `YY.MM.mm`, where `YY` is the last two digits of the year of the
     72release, `MM` is the number of the release in the year, and `mm` is the
     73number of the patch release.
     74
     75For example:
     76
     77* The first release of 2010 is `10.0.0`
     78
     79* The second release of 2010 is `10.1.0`
     80
     81* If 10.1.0 has some critical defects, then a patch release would be
     82  numbered `10.1.1`
     83
     84* The first pre-release of `10.0.0` is `10.0.0pre1`, the second is `10.0.0pre2`
     85
     86Every release of Twisted includes the whole project, the core and all
     87sub-projects. Each of these has the same number.
     88
     89Throughout this document, we'll refer to the version number of the
     90release as `$RELEASE`. Examples of `$RELEASE` include `10.0.0`,
     91`10.1.0`, `10.1.1` etc.
     92
     93We'll refer to the first two components of the release as `$API`, since
     94all releases that share those numbers are mutually API compatible. e.g.
     95for `10.0.0`, `$API` is `10.0`; for `10.1.0` and `10.1.1`, `$API` is
     96`10.1`.
     97
     98The change-versions script automatically picks the right number for you.
     99Please retrieve it after you run it.
     100
     101
     102Overview
     103--------
     104
     105To release Twisted, we
     106
     1071. Prepare for a release
     108
     1092. Release N pre-releases
     110
     1113. Release the final release
     112
     113
     114Prepare for a release
     115---------------------
     116
     1171. Check the milestone for the upcoming release
     118
     119   1. Get rid of any non-critical bugs
     120
     121   2. Get any critical bugs fixed
     122
     123   3. Check the `release manager notes`_ in case anyone has left
     124      anything which can only be done during the release.
     125
     1262. Check for any `regressions`_
     127
     1283. Read through the `INSTALL` and `README` files to make sure things like the
     129   supported Python versions are correct
     130
     131   1. Check the required Python version.
     132
     133   2. Check that the list matches the current set of buildbots.
     134
     135   3. Any mistakes should be fixed in trunk before making the release branch
     136
     1374. Choose a version number. Remember to include the 'preN' suffix
     138
     1395. File a ticket
     140
     141   1. Assign it to the upcoming release milestone
     142
     143   2. Assign it to yourself
     144
     145   3. Call it "Release $RELEASE"
     146
     1476. Make a branch (e.g. `mkbranch Twisted releases/release-$RELEASE-4290`)
     148
     149.. _release manager notes: https://twistedmatrix.com/trac/wiki/ReleaseManagerNotes
     150.. _regressions: http://twistedmatrix.com/trac/query?status=new&status=assigned&status=reopened&type=regression&order=priority
     151
     152
     153How to do a pre-release
     154-----------------------
     155
     1561. Check  buildbot to make sure all supported platforms are green (wait
     157   for pending builds if necessary).
     158
     1592. If a previously supported platform does not currently have a
     160   buildbot, move from supported platforms to "expected to work" in
     161   `INSTALL`. (Pending `#1305`_)
     162
     1633. Run `./bin/admin/change-versions --prerelease`
     164
     1654. Commit the changes made by change-versions
     166
     1675. Run `./bin/admin/build-news .`
     168
     1696. Commit the changes made by build-news - this automatically removes
     170   the NEWS topfiles (see `#4315`_)
     171
     1727. Bump copyright dates in `LICENSE`, `twisted/copyright.py`, and `README` if
     173   required
     174
     1758. Make a temporary directory for the tarballs to live in (e.g. `mkdir
     176   /tmp/twisted-release`)
     177
     1789. Run `./bin/admin/build-tarballs . /tmp/twisted-release/`
     179
     180.. note:: build-tarballs does not produce exactly the same output
     181   when run multiple times, even when nothing else has changed.
     182   If a problem is encountered that requires build-tarballs to
     183   be re-run (either during the pre-release or later during the
     184   release), care must be taken to avoid releasing two or more
     185   **different** versions of the tarball.
     186
     18710. Copy `NEWS` to `/tmp/twisted-release/` as `NEWS.txt` for people to view
     188    without having to download the tarballs.
     189
     190   1. `cp NEWS /tmp/twisted-release/NEWS.txt`
     191
     19211. Upload the tarballs to twistedmatrix.com/Releases/pre/$RELEASE (see
     193     `#4353`_)
     194
     195   1. You can use
     196      `rsync --rsh=ssh --partial --progress -av
     197      /tmp/twisted-release/t-web@dornkirk.twistedmatrix.com:/srv/t-web/data/
     198      releases/pre/<RELEASE>/`
     199      to do this.
     200
     20112. Write the pre-release announcement
     202
     203   1. Read through the NEWS file and summarize the interesting changes for the
     204      release
     205
     206   2. Get someone else to look over the announcement before doing it
     207
     20813. Announce the pre-release on
     209
     210   1. the twisted-python `mailing list`_
     211
     212   2. on IRC in the `#twisted` topic
     213
     214   3. in a blog post, ideally labs.twistedmatrix.com
     215
     216.. _#1305: https://twistedmatrix.com/trac/ticket/1305
     217.. _#4315: https://twistedmatrix.com/trac/ticket/4315
     218.. _#4353: https://twistedmatrix.com/trac/ticket/4353
     219.. _mailing list: https://twistedmatrix.com/trac/wiki/TwistedCommunity#MailLists
     220
     221
     222Pre-release announcement
     223~~~~~~~~~~~~~~~~~~~~~~~~
     224
     225The pre-release announcement should mention the important changes since the
     226last release, and exhort readers to test this pre-release.
     227
     228Here's what the $RELEASEpre1 release announcement might look like::
     229
     230    Live from PyCon Atlanta, I'm pleased to herald the approaching
     231    footsteps of the $API release.
     232   
     233    Tarballs for the first Twisted $RELEASE pre-release are now available at:
     234     http://people.canonical.com/~jml/Twisted/
     235   
     236    Highlights include:
     237   
     238     * Improved documentation, including "Twisted Web in 60 seconds"
     239   
     240     * Faster Perspective Broker applications
     241   
     242     * A new Windows installer that ships without zope.interface
     243   
     244     * Twisted no longer supports Python 2.3
     245   
     246     * Over one hundred closed tickets
     247   
     248    For more information, see the NEWS file.
     249   
     250    Please download the tarballs and test them as much as possible.
     251   
     252    Thanks,
     253    jml
     254
     255How to do a final release
     256-------------------------
     257
     258
     259Prepare the branch
     260~~~~~~~~~~~~~~~~~~
     261
     262
     2631. Have the release branch, previously used to generate a pre-release,
     264   checked out
     265
     2662. Run `./bin/admin/change-versions`
     267
     2683. Add the quote of the release to the `README`
     269
     2704. Make a new quote file for the next version: `svn cp
     271   docs/fun/Twisted.Quotes docs/historic/Quotes/Twisted-$API; echo '' >
     272   docs/fun/Twisted.Quotes`.
     273
     2745. Commit the version and `README` changes.
     275
     2766. Submit the ticket for review
     277
     2787. Pause until the ticket is reviewed and accepted.
     279
     2808. Tag the release
     281
     282   * e.g. `svn cp svn+ssh://svn.twistedmatrix.com/svn/Twisted/branches/releases/release-$RELEASE-4290
     283     svn+ssh://svn.twistedmatrix.com/svn/Twisted/tags/releases/twisted-$RELEASE`
     284
     285   * A good commit message to use is something like "Tag $RELEASE
     286     release"
     287
     288
     289Cut the tarballs & installers
     290~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     291
     2921. Create a new staging area for the release (e.g. `mkdir /tmp/twisted-
     293   release`)
     294
     2952. Using a checkout of the release branch or the release tag (with no
     296   local changes!), run `./bin/admin/build-tarballs . /tmp/twisted-release/`
     297
     2983. Build windows installers
     299
     300   1. http://buildbot.twistedmatrix.com/builders/winxp32-py2.7-msi/
     301
     302   2. http://buildbot.twistedmatrix.com/builders/windows7-64-py2.7-msi
     303
     304   3. For "Branch" specify the release branch, e.g.
     305      "branches/releases/release-$RELEASE-4290"
     306
     307   4. Download the latest .msi, .exe and .whl files from from
     308      http://buildbot.twistedmatrix.com/builds/twisted-packages/ and save
     309      them in the staging directory
     310
     3114. Sign the tarballs and Windows installers. (You will need a PGP key
     312   for this - use something like Seahorse to generate one, if you don't
     313   have one.)
     314
     315   1. MD5: `md5sum Tw* | gpg -a --clearsign > /tmp/twisted-release/twisted
     316      -$RELEASE-md5sums.txt`
     317
     318   2. SHA512: `shasum -a 512 Tw* | gpg -a --clearsign > /tmp/twisted-
     319      release/twisted-$RELEASE-shasums.txt`
     320
     321   3. Compare these to an `example of twisted-$RELEASE-md5sums.txt`_ -
     322      they should look the same.
     323
     324.. _example of twisted-$RELEASE-md5sums.txt: http://tmrc.mit.edu/mirror/twisted/twisted-10.1.0-md5sums.txt
     325
     326
     327Update documentation
     328~~~~~~~~~~~~~~~~~~~~
     329
     3301. Get the dependencies
     331
     332   * Pydoctor (use the version from `lp:~twisted-dev/pydoctor/twisted-templates-2`_)
     333
     334   * `Nevow`_
     335
     336   * Epydoc (python-epydoc in Debian)
     337
     338   * Some version of Latex if you didn't use the Epydoc Debian/Ubuntu
     339     package or ignored the recommended packages
     340
     341   * netpbm (graphics conversion tools used by the book-builder)
     342
     3432. In another directory, export the version of Twisted being released
     344
     345   * e.g. `svn export svn+ssh://svn.twistedmatrix.com/svn/Twisted/tags/releases/twisted-$RELEASE`
     346
     3473. Build the documentation
     348
     349   1. `cd twisted-$RELEASE`
     350
     351   2. Retrieve the template file (eg. `wget https://raw.github.com/twisted-infra/t-web/master/docs/website-template.tpl`)
     352
     353   3. `./bin/admin/build-docs . ./website-template.tpl`
     354
     355      * If you have latex errors when trying to build the book and retry,
     356        don't forget to clean tmp directories which may be leftover in
     357        doc/core/howto.
     358
     359   4. `cp -R doc /tmp/twisted-release/`
     360
     3614. Run the build-apidocs script to build the API docs (in an export of
     362   the release tag) and then upload them (See also `APIDocs`_ and
     363   `#2891`_).
     364
     365   1. Copy the pydoctor and www directories from twisted-templates-2 into
     366      your export.
     367
     368   2. `./bin/admin/build-apidocs . /tmp/twisted-release/api`
     369
     370   3. Documentation will be generated in a directory called /tmp/twisted-release/api
     371
     372.. _lp:~twisted-dev/pydoctor/twisted-templates-2: https://code.launchpad.net/~twisted-dev/pydoctor/twisted-templates-2
     373.. _Nevow: https://launchpad.net/divmod.org
     374.. _APIDocs: https://twistedmatrix.com/trac/wiki/APIDocs
     375.. _#2891: https://twistedmatrix.com/trac/ticket/2891
     376
     377
     378Distribute
     379~~~~~~~~~~
     380
     381
     3821. Create a tarball with the contents of the release directory: `cd
     383   /tmp/twisted-release; tar -cvjf ../release.tar.bz2 *`
     384
     3852. Upload to the official upload locations (see `#2888`_)
     386
     387   1. `cd ~; git clone https://github.com/twisted-infra/braid`
     388
     389   2. `cd braid; git submodule update --init`
     390
     391   3. `cd services/t-web/; git checkout release-process; cd ../../`
     392
     393   4. `virtualenv ~/dev/braid; source ~/dev/braid/bin/activate; cd
     394      ~/braid; python setup.py develop;`
     395
     396   5. `cd ~/braid; fab config.production
     397      t-web.uploadRelease:$RELEASE,/tmp/release.tar.bz2`
     398
     3993. Test the generated docs
     400
     401   1. Browse to http://twistedmatrix.com/documents/$RELEASE/
     402
     403   2. Make sure that there is content in each of the directories and that
     404      it looks good
     405
     406   3. Follow each link on
     407      http://twistedmatrix.com/trac/wiki/Documentation, replace current
     408      with $RELEASE (e.g. `10.0.0`) and look for any obvious breakage
     409
     4104. Change the "current" symlink
     411
     412   1. Upload release: `fab config.production t-web.updateCurrentDocumentation:$RELEASE`
     413
     414.. _#2888: https://twistedmatrix.com/trac/ticket/2888
     415
     416
     417Announce
     418~~~~~~~~
     419
     4201. Update `Downloads`_ pages
     421
     422   1. The following updates are automatic, due to the use of the
     423      `ProjectVersion`_ `wiki macro`_ throughout most of the `Downloads`_
     424      page.
     425
     426      1. Text references to the old version to refer to the new version
     427
     428      2. The link to the NEWS file to point to the new version
     429
     430      3. Links and text to MSIs
     431
     432      4. Links and text to the main tarball
     433
     434      5. Links and text to sub tarballs, including links to news files
     435
     436   2. Add a new md5sum link
     437
     438   3. Add a new shasum link
     439
     440   4. Save the page, check all links
     441
     4422. Update PyPI records & upload files
     443
     444   * http://pypi.python.org/pypi/Twisted/
     445
     446     * Edit the version. *Make sure you do this first.*
     447
     448     * Upload tarball, MSIs and wheels
     449
     4503. Write the release announcement (see below)
     451
     4524. Update `ReleaseRevisions`_ with the revision of this release
     453
     454   * The final revision of the release is the revision in which the
     455     release branch was made. The starting revision is the one immediately
     456     after the final revision of the previous release.
     457
     4585. Announce the release
     459
     460   1. Send a text version of the announcement to: `twisted-
     461      python@twistedmatrix.com, python-announce-list@python.org, python-
     462      list@python.org, twisted-web@twistedmatrix.com, twisted-jabber@ik.nu`
     463
     464      * Note: you need to be subscribed to some of those lists to be able to
     465        send, like `twisted-jabber@ik.nu`.
     466
     467   2. Launchpad;
     468
     469      * https://launchpad.net/twisted/main/+addrelease
     470
     471        * Include a text version of the announcement and the new entries
     472          of the `NEWS` file
     473
     474      * https://launchpad.net/twisted/+announcements
     475
     476   3. http://labs.twistedmatrix.com
     477
     478      * Post a web version of the announcements, with links instead of
     479        literal URLs
     480
     481   4. Twitter, if you feel like it
     482
     483   5. `#twisted` topic on IRC (you'll need ops)
     484
     4856. Merge the release branch into trunk, closing the release ticket at
     486   the same time.
     487
     488   * For now you need to add a .misc `NEWS` fragment to merge the branch.
     489
     4907. Close the release milestone (which should have no tickets in it).
     491
     4928. Open a milestone for the next release.
     493
     494.. _Downloads: https://twistedmatrix.com/trac/wiki/Downloads
     495.. _ProjectVersion: https://raw.github.com/twisted-infra/twisted-trac-plugins/master/twisted_trac_plugins/release_macro.py
     496.. _wiki macro: https://twistedmatrix.com/trac/wiki/WikiMacros
     497.. _ReleaseRevisions: https://twistedmatrix.com/trac/wiki/ReleaseRevisions
     498
     499
     500Release announcement
     501~~~~~~~~~~~~~~~~~~~~
     502
     503The final release announcement should:
     504
     505* Mention the version number
     506
     507* Include links to where the release can be downloaded
     508 
     509* Summarize the significant changes in the release
     510
     511* Consider including the quote of the release
     512 
     513* Thank the contributors to the release
     514
     515
     516Here's an example:
     517
     518::
     519
     520    On behalf of Twisted Matrix Laboratories, I am honoured to announce
     521    the release of Twisted 13.2!
     522   
     523    The highlights of this release are:
     524   
     525     * Twisted now includes a HostnameEndpoint implementation which uses
     526    IPv4 and IPv6 in parallel, speeding up the connection by using
     527    whichever connects first (the 'Happy Eyeballs'/RFC 6555 algorithm).
     528    (#4859)
     529   
     530     * Improved support for Cancellable Deferreds by kaizhang, our GSoC
     531    student. (#4320, #6532, #6572, #6639)
     532   
     533     * Improved Twisted.Mail documentation by shira, our Outreach Program
     534    for Women intern. (#6649, #6652)
     535   
     536     * twistd now waits for the application to start successfully before
     537    exiting after daemonization. (#823)
     538   
     539     * SSL server endpoint string descriptions now support the
     540    specification of chain certificates. (#6499)
     541   
     542     * Over 70 closed tickets since 13.1.0.
     543   
     544    For more information, check the NEWS file (link provided below).
     545   
     546    You can find the downloads at <https://pypi.python.org/pypi/Twisted>
     547    (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>) .
     548    The NEWS file is also available at
     549    <http://twistedmatrix.com/Releases/Twisted/13.2/NEWS.txt>.
     550   
     551    Many thanks to everyone who had a part in this release - the
     552    supporters of the Twisted Software Foundation, the developers who
     553    contributed code as well as documentation, and all the people building
     554    great things with Twisted!
     555   
     556    Twisted Regards,
     557    HawkOwl
     558
     559
     560When things go wrong
     561--------------------
     562
     563If you discover a showstopper bug during the release process, you have
     564three options.
     565
     5661. Abort the release, make a new point release (e.g. abort `10.0.0`,
     567   make `10.0.1` after the bug is fixed)
     568
     5692. Abort the release, make a new pre-release (e.g. abort `10.0.0`, make
     570   `10.0.0pre3` after the bug is fixed)
     571
     5723. Interrupt the release, fix the bug, then continue with it (e.g.
     573   release `10.0.0` with the bug fix)
     574
     575
     576If you choose the third option, then you should:
     577
     5781. Delete the tag for the release
     579
     5802. Recreate the tag from the release branch once the fix has been
     581   applied to that branch
     582
     583
     584Examples
     585--------
     586
     587
     588Open questions
     589--------------
     590
     591* How do we manage the case where there are untested builds in trunk?
     592
     593* Should picking a release quote be part of the release or the pre-
     594  release?
     595
     596* What bugs should be considered release blockers?
     597
     598  * Ultimately it's the RM's discretion
     599
     600* Should news fragments contain information about who made the
     601  changes?
     602
     603* A thought for future releases: since we'd really like folks to
     604  download the prereleases and try them out, perhaps we should put the
     605  `NEWS`_ file on the web somewhere official, too, so they can see all
     606  the cool stuff they can try out?
     607
     608  * **XXX**: jml doesn't know what this means any more
     609
     610.. _NEWS: http://twistedmatrix.com/trac/browser/trunk/NEWS
     611
     612
     613Bugs mentioned on this page
     614---------------------------
     615
     616* `Automate uploading tarballs to TMRC`_
     617
     618* `Automate uploading pre-release tarballs`_
     619
     620* `Convenience command for removing news fragments`_
     621
     622* `Automate building & uploading API docs for website`_
     623
     624* `Automate building & uploading howto docs for website`_
     625
     626* `Lore applies quoting to contents of script tags`_
     627
     628* `Lore collapses script tags`_
     629
     630.. _Automate building & uploading API docs for website: https://twistedmatrix.com/trac/ticket/2891
     631.. _Automate building & uploading howto docs for website: https://twistedmatrix.com/trac/ticket/2380
     632.. _Automate uploading tarballs to TMRC: https://twistedmatrix.com/trac/ticket/2888
     633.. _Automate uploading pre-release tarballs: https://twistedmatrix.com/trac/ticket/4353
     634.. _Convenience command for removing news fragments: https://twistedmatrix.com/trac/ticket/4315
     635.. _Lore applies quoting to contents of script tags: https://twistedmatrix.com/trac/ticket/4544
     636.. _Lore collapses script tags: https://twistedmatrix.com/trac/ticket/4545
     637
     638
     639See also
     640--------
     641
     642
     643* `release management tickets`_
     644
     645* `regular-release tickets`_
     646
     647* `ReleaseAutomation`_
     648
     649* `Releasing Bazaar`_ -- release documentation for another project
     650  that does time-based releases
     651
     652.. _release management tickets: http://twistedmatrix.com/trac/query?status=new&status=assigned&status=reopened&component=release+management&order=priority
     653.. _regular-release tickets: http://twistedmatrix.com/trac/query?status=new&status=assigned&status=reopened&milestone=regular-releases&order=priority
     654.. _ReleaseAutomation: https://twistedmatrix.com/trac/wiki/ReleaseAutomation
     655.. _Releasing Bazaar: http://doc.bazaar.canonical.com/bzr.dev/developers/releasing.html