Twisted Compatibility Policy
============================
Motivation
----------
The Twisted project has a small development team, and we cannot afford to provide anything but critical bug-fix support for multiple version branches of Twisted.
However, we all want Twisted to provide a positive experience during development, deployment, and usage.
Therefore we need to provide the most trouble-free upgrade process possible, so that Twisted application developers will not shy away from upgrades that include necessary bugfixes and feature enhancements.
Twisted is used by a wide variety of applications, many of which are proprietary or otherwise inaccessible to the Twisted development team.
Each of these applications is developed against a particular version of Twisted.
The most important compatibility to preserve is at the Python API level.
Python does not provide us with a strict way to partition **public** and **private** objects (methods, classes, modules), so it is unfortunately quite likely that many of those applications are using arbitrary parts of Twisted.
Our compatibility strategy needs to take this into account, and be comprehensive across our entire codebase.
Exceptions can be made for modules aggressively marked **unstable** or **experimental**, but even experimental modules will start being used in production code if they have been around for long enough.
The purpose of this document is to to lay out rules for Twisted application developers who wish to weather the changes when Twisted upgrades, and procedures for Twisted engine developers - both contributors and core team members - to follow when who want to make changes which may be incompatible to Twisted itself.
Defining Compatibility
----------------------
The word "compatibility" is itself difficult to define.
While comprehensive compatibility is good, total compatibility is neither feasible nor desirable.
Total compatibility requires that nothing ever change, since any change to Python code is detectable by a sufficiently determined program.
There is some folk knowledge around which kind of changes **obviously** won't break other programs, but this knowledge is spotty and inconsistent.
Rather than attempt to disallow specific kinds of changes, here we will lay out a list of changes which are considered compatible.
Throughout this document, **compatible** changes are those which meet these specific criteria.
Although a change may be broadly considered backward compatible, as long as it does not meet this official standard, it will be officially deemed **incompatible** and put through the process for incompatible changes.
The compatibility policy described here is 99% about changes to **interface**,
not changes to functionality.
.. note::
Ultimately we want to make the user happy but we cannot put every possible thing that will make every possible user happy into this policy.
Brief notes for developers
--------------------------
Here is a summary of the things that need to be done for deprecating code.
This is not an exhaustive read and beside this list you should continue reading the rest of this document:
* Do not change the function's behavior as part of the deprecation process.
* Cause imports or usage of the class/function/method to emit a DeprecationWarning either call warnings.warn or use one of the helper APIs
* The warning text must include the version of Twisted in which the function is first deprecated (which will always be a version in the future)
* The warning text should recommend a replacement, if one exists.
* The warning must "point to" the code which called the function. For example, in the normal case, this means stacklevel=2 passed to warnings.warn.
* There must be a unit test which verifies the deprecation warning.
* A .removal news fragment must be added to announce the deprecation.
Procedure for Incompatible Changes
----------------------------------
Any change specifically described in the next section as **compatible** may be made at any time, in any release.
The First One's Always Free
^^^^^^^^^^^^^^^^^^^^^^^^^^^
The general purpose of this document is to provide a pleasant upgrade experience for Twisted application developers and users.
The specific purpose of this procedure is to achieve that experience by making sure that any application which runs without warnings may be upgraded one minor version of twisted (y to y+1 in x.y.z) or from the last minor revision of a major release to the first minor revision of the next major release (x to x + 1 in x.y.z to x.0.z, when there will be no x.y+1.z).
In other words, any application which runs its tests without triggering any warnings from Twisted should be able to have its Twisted version upgraded at least once with no ill effects except the possible production of new warnings.
Incompatible Changes
^^^^^^^^^^^^^^^^^^^^
Any change which is not specifically described as **compatible** must be made in 2 phases.
If a change is made in release R, the timeline is:
1. Release R: New functionality is added and old functionality is deprecated with a DeprecationWarning.
2. At the earliest, release R+2 and one year after release R, but often much later: Old functionality is completely removed.
Removal should happen once the deprecated API becomes an additional maintenance burden.
For example, if it makes implementation of a new feature more difficult, if it makes documentation of non-deprecated APIs more confusing, or if its unit tests become an undue burden on the continuous integration system.
Removal should not be undertaken just to follow a timeline. Twisted should strive, as much as practical, not to break applications relying on it.
Procedure for Exceptions to this Policy
---------------------------------------
**Every change is unique.**
Sometimes, we'll want to make a change that fits with the spirit of this document (keeping Twisted working for applications which rely upon it) but may not fit with the letter of the procedure described above (the change modifies behavior of an existing API sufficiently that something might break).
Generally, the reason that one would want to do this is to give applications a performance enhancement or bug fix that could break behavior in unintended hypothetical uses of an existing API, but we don't want well-behaved applications to pay the penalty of a deprecation/adopt-a-new-API/removal cycle in order to get the benefits of the improvement if they don't need to.
If this is the case for your change, it's possible to make such a modification without a deprecation/removal cycle.
However, we must give users an opportunity to discover whether a particular incompatible change affects them: we should not trust our own assessments of how code uses the API.
In order to propose an incompatible change, start a discussion on the mailing list.
Make sure that it is eye-catching, so those who don't read all list messages in depth will notice it, by prefixing the subject with **INCOMPATIBLE CHANGE:** (capitalized like so).
Always include a link to the ticket, and branch (if relevant).
In order to **conclude** such a discussion, there must be a branch available so that developers can run their unit tests against it to mechanically verify that their understanding of their own code is correct.
If nobody can produce a failing test or broken application within **a week's time** from such a branch being both 1. available and 2. announced, and at least **three committers** agree that the change is worthwhile, then the branch can be considered approved for the incompatible change in question.
Since some codebases that use Twisted are presumably proprietary and confidential, there should be a good-faith presumption if someone says they have broken tests but cannot immediately produce code to share.
The branch must be available for one week's time.
.. note::
The announcement forum for incompatible changes and the waiting period required are subject to change as we discover how effective this method is; the important aspect of this policy is that users have some way of finding out in advance about changes which might affect them.
Compatible Changes. Changes not Covered by the Compatibility Policy
-------------------------------------------------------------------
Here is a non-exhaustive list of changes which are not covered by the compatibility policy.
These changes can be made without having to worry about the compatibility policy.
Test Changes
^^^^^^^^^^^^
No code or data in a test package should be imported or used by a non-test package within Twisted.
By doing so, there's no chance anything could access these objects by going through the public API.
Test code and test helpers are considered private API and should not be imported outside
of the Twisted testing infrastructure.
Private Changes
^^^^^^^^^^^^^^^
Code is considered *private* if the user would have to type a leading underscore to access it.
In other words, a function, module, method, attribute or class whose name begins with an underscore may be arbitrarily changed.
Bug Fixes and Gross Violation of Specifications
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If Twisted documents an object as complying with a published specification, and there are inputs which can cause Twisted to behave in obvious violation of that specification, then changes may be made to correct the behavior in the face of those inputs.
If application code must support multiple versions of Twisted, and work around violations of such specifications, then it must test for the presence of such a bug before compensating for it.
For example, Twisted supplies a DOM implementation in twisted.web.microdom.
If an issue were discovered where parsing the string `Hello` and then serializing it again resulted in `>xml/xml<`, that would grossly violate the XML specification for well-formedness.
Such code could be fixed with no warning other than release notes detailing that this error is now fixed.
Raw Source Code
^^^^^^^^^^^^^^^
The most basic thing that can happen between Twisted versions, of course, is that the code may change.
That means that no application may ever rely on, for example, the value of any **func_code** object's **co_code** attribute remaining stable, or the **checksum** of a .py file remaining stable.
**Docstrings** may also change at any time.
Applications must not depend on any Twisted class, module, or method's metadata attributes such as ``__module__``, ``__name__``, ``__qualname__``, ``__annotations__`` and ``__doc__`` to remain the same.
New Attributes
^^^^^^^^^^^^^^
New code may also be added.
Applications must not depend on the output of the ``dir()`` function on any object remaining stable, nor on any object's ``__all__`` attribute, nor on any object's ``__dict__`` not having new keys added to it.
These may happen in any maintenance or bugfix release, no matter how minor.
Pickling
^^^^^^^^
Even though Python objects can be pickled and unpickled without explicit support for this, whether a particular pickled object can be unpickled after any particular change to the implementation of that object is less certain.
Because of this, applications must not depend on any object defined by Twisted to provide pickle compatibility between any release unless the object explicitly documents this as a feature it has.
Representations
^^^^^^^^^^^^^^^
The printable representations of objects, as returned by ``repr(