[Twisted-Python] The Twisted 14.0 Release Pre-Post-Mortem, and Where To From Here

[HawkOwl]

Hi everyone,

I’m sure that some of you have been following the past seven or so weeks of Twisted 14.0 release shenanigans, and this email hopes to explain what went wrong, what we can do better next time, and where we can go from here.

Problem 1: Twisted 14.0.0pre1 had a regression. This was not noticed in the prerelease stage because it was not marked as a regression, where the RM does a check for open regressions on the milestone.
What we can do better next time: Tickets that are regressions need to be marked as regressions and applied to the release milestone. If you think it might be a regression - even slightly - mark it as such, and comment that you are not sure. It’s easier to find the ticket later and decide it is not actually a regression than have to abort a release because it’s come up after a prerelease.

Problem 2: The fix for the regression was not merged into pre1, the release was rerolled from trunk. This meant some pyOpenSSL and TLS improvements got into the 14.0 release from pre2 onwards, but introduced new regressions.
What we can do better next time: Do not reroll from trunk to get bug fixes - merge them into the release branch. 

Problem 3: The fixes for the regressions were finished after some delay, since the fixes had to be written and reviewed. This introduced delays into the 14.0 release cycle.
What we can do better next time: Rather than fix regressions introduced, the ticket that introduced them should be reverted.

Problem 4: The fixes for the regressions did not merge cleanly with the release branch. Some 35+ tickets were merged between pre1 and the release of the regression fix into trunk.
What we can do better next time: Bug fixes should be based off the release branch, not trunk. This reduces the likelihood of code churn or unknown dependencies causing problems during the merge.

Problem 5: There was mixed communication whether one of the regression fixes was to be introduced in 14.0 or in a bug fix release (14.0.1).
What we can do better: If a fix is intended for merging in to a prerelease, it should be raised on the mailing list, so that there is more visibility for its intentions.

Problem 6: I personally made several mistakes along the way - from screwing up svn merges to interpreting the “abort the release and incorporate the bugfix” to apply the initial regression fix. Since the TLS changes were topical, I decided that having them out ASAP would be better than not.
What we can do better: Improved docs/automation to reduce the margin for RM error, and better automation to make a new release to get out important features really easy.

These are the major problems which I have identified - I’m sure there’s plenty more, and I would like people to list them if I have not - even if they make me look like an idiot ;). We can learn from it, I’m sure.

So, this leaves where to from now. I see a few options, with my estimates for work and risk that it’ll explode:

1 - Most work, high risk - Work on making the regression fixes merge cleanly with 14.0.0pre5. This is big-ish task with room for error, since there was some underlying code churn.
2 - Some work, medium risk - Release 14.0.0pre5 as 14.0 final, and I (or another RM if I’m no longer trusted ;) ) initiate the 14.1 release immediately.
3 - Least work, highish risk - Scrap 14.0, begin the 14.1 release immediately. since 14.0 tags become 14.1 tags, and we just have to hope that there’s no regressions in the 39 tickets fixed between pre1 and now. This may introduce issues (since 14.0 is an un-release, and there are questions about what this does to our deprecation windows).

If I am to be honest, I much prefer option #3, but I would like opinions from other developers, before I go causing more problems than I already have :)

Regards,
HawkOwl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140507/ce265795/attachment.pgp>