[Twisted-Python] [RFC] Deprecation Policy

[glyph at divmod.com]

On 18 Apr, 05:06 am, jml at mumak.net wrote:
>Hello everyone,
>
>Now that the release is out[1], I'd like to start work on fixing #1216
>-- official deprecation policy.

Sorry for the delay.  Due to a miscommunication, I thought that Chris 
would be the one writing up our meeting notes and announcing this. 
After I realized I was the one who was supposed to be doing it, I wanted 
to write up something official-sounding on the wiki first, but of 
course, there's never enough time.  So here's a rough cut at the policy 
that we (the current "TSF board", such as it is: chris, me, JP, and 
itamar) agreed on at the sprint in Cambridge.

Thanks, Jonathan, for motivating this; we've needed a clear statement 
for a long time.  Someone (and this will probably be me) needs to make a 
complete and unambiguous index of official Twisted policy and put it 
together somewhere, probably on the wiki, at least at first.
>I think these are all the questions we need to answer:

The proposed policy does break down as a list of answers to these 
questions, so one point at a time:
>- Is it time-based or release-based?

Both.  There will be a minimum amount of time and a minimum number of 
releases before a deprecated feature may be removed.
>- How long for?

Every deprecation must last for a minimum of 2 releases and a minimum of 
one year.  Of course, anyone who wants to keep maintaining deprecated 
APIs for longer than that may do so.

I know this is somewhat longer than you've suggested, but at the meeting 
we ran down a laundry list of real-world usages of Twisted and pretty 
much nobody upgrades Twisted more than once per year.  In fact, once per 
2 years seems to be the average, and that's a fairly ambitious estimate. 
The objective of providing a year-long deprecation period is to allow a 
user who is performing their bi-yearly Twisted upgrade to do so with no 
more than 2 intermediary upgrades.
>- Do we grade deprecations? How?

There will be three phases of deprecation.  When a deprecation is first 
introduced (in trunk) the "deprecation counter" will be at 3.  When that 
deprecation sees its first release, it decrements to 2.  Deprecations at 
levels 3 and 2 will be PendingDeprecationWarning.  When that same 
deprecation sees its second release after a 6-month period, the counter 
decrements to 1, and finally, after a second 6-month period, it 
decrements to 0 and can be removed.  Deprecations with a counter at 1 
and at 0 (if they still exist) should emit a DeprecationWarning.

The counter may not move down past 2 until *all instances of the emitted 
warning* have been removed from the Twisted test suite, with the 
exception of tests specifically for the deprecated functionality itself 
(i.e. tests that call assertWarns).

The reason that this is described as a counter rather than primarily in 
terms of (Pending)DeprecationWarning is that this was a subject of 
intense debate during the meeting.  There is definitely a consensus that 
the Python warnings system is not a sufficient tool we have to deal with 
the kind of notifications that we want to provide.  There is also a 
consensus that it is the only tool available.  So, let me provide some 
more detail on the desired behavior here.

The idea behind using PendingDeprecationWarning is that *users* of a 
piece of software which depends on Twisted should, in principle, be able 
to upgrade Twisted by one version without seeing any error output. 
However, *maintainers* of that software should immediately see all the 
warnings when they test with a new release.  Trial should change to show 
all PendingDeprecationWarnings by default.  However, this might be 
impractical due to other uses of PendingDeprecationWarning - if that is 
the case, Twisted should include its own warning class.  It may be 
helpful to do that in order to include the additional metadata required 
for an automated deprecation workflow, however.  The desired deprecation 
workflow is:

  * Twisted developer deprecates functionality, by adding a 
warnings.warn(TwistedDeprecationWarning(some useful metadata)) to the 
code.

  * The release process automatically runs "update-deprecations" over the 
codebase, updating the deprecation counters in the developer-supplied 
metadata appropriately.  The release goes out.  (Ideally this would be 
done without actually having to modify the whole codebase, updating a 
central deprecation / release index.)

  * Another release happens that sets some counters to zero.

  * A developer drops support for some deprecated functionality, 
confident in their ability to do so by observing the zero (or negative) 
value for the counter.

An obvious remaining piece of work to implement this final policy is to 
define some temporary manual things that we can do until such a tool is 
implemented, and implement the tool itself as soon as possible.  I don't 
believe any tickets have yet been filed for this, and *that* I am 
leaving in our venerable release manager's court, since this is all a 
part of the release process except for the fairly minor change to trial.

However, in the absence of any tools, this policy does apply now, it 
will just be a bit of a pain to work out where the counter stands on any 
particular deprecation until we've got some more metadata to help us.
>- Which code is subject to this policy?

All Python code released in Twisted, with the exceptions of private 
modules and test modules.

Only code which has been present in a release needs to be deprecated. 
Anything which was added after the last release and has not yet been 
present in any other release may be modified at will.  (We try to keep 
trunk stable, but this only applies to code that was relying on features 
in previous releases.)  I think that CompatibilityPolicy's explanation 
of an "incompatible change" explains this.

Users: do not import anything from a package named "test", or any name 
(module, class, or function) which begins with "_"; bears will eat you. 
This code may change at any time.

(Not a part of the policy: I personally want to add some caveats here, 
where users of Twisted can ask for a grace period on private APIs that 
people actually needed to use, but I think for now let's err on the side 
of allowing the deprecation - I don't want to encourage people to use 
private APIs.  Users: if you find that there are things you can't do 
with the public APIs and you really need a grace period to upgrade your 
applications, please speak up.  Eventually, I'd like to have a very 
aggressive "you can't even import (or invoke) this if it's private" 
regime where users can't accidentally use private functionality without 
first seeing a runtime error of some kind that they have to explicitly 
silence, as with my "pyexport" hack.  Once we've done *that* we should 
definitely have a zero-tolerance policy on supporting private stuff, 
though...)

There's obviously some work to do here to make this policy easy and 
convenient to follow.  This needs to be edited, CompatibilityPolicy 
updated, and prominently linked.  We need someone to write up at least 
two different summaries: one for maintainers of projects using Twisted 
("What kind of compatibility should I expect from Twisted releases?") 
one for Twisted contributors ("How do I add a deprecation?  When can I 
remove it?") and maybe one for users and one for packagers, too.  We 
need some tools to help us update deprecations.  We need trial to change 
so that developers are notified earlier than their users.

Do I hear any volunteers...? :)