== Objective ==

To have Twisted available on Python 3.3 and newer, with the same functionality as is currently available on Python 2.x.

== Strategy ==

  * Single source - The result will be a version of Twisted that is source-compatible with Python 2.6, Python 2.7, and Python 3.3.  Please note that this specifically means you '''should not use 2to3 at any point in this process'''.
  * Test-driven - Following the standard Twisted development process, all changed code will be fully automatically unit tested.
  * Opportunistic - Where module dependencies (within Twisted) need to be ported to support porting of the high-priority modules, subsets of the module may be ported to provide specific functionality (but less than the full module).

All other standard Twisted development practices apply.

An initial minimal project (http://twistedmatrix.com/trac/milestone/Python%203.3%20Minimal) was funded by Canonical and implemented by Future Foundries LLC (i.e. Itamar and Jean-Paul), with the aim of getting a basic subset of Twisted working, sufficient to run a specific Canonical project.  That project has been completed, laying the groundwork for further porting effort by other developers.

== Reviewer check list ==

  1. Tests pass under 3.3.
  1. Ported modules were added to list of ported modules - [source:trunk/admin/_twistedpython3.py admin/_twistedpython3.py]
  1. No warnings when running using `-3` mode under Python 2.7.
  1. All non-docstring strings are clearly marked as one of the various string categories (bytes/unicode/identifier/version-dependent).
  1. No use of 'str' except for things that really can differ across versions of Python.
  1. `from __future__ import division, absolute_import` was added to the module.
    1. And any code changes necessary to work with the resulting changed semantics were made.
  1. Verify dictionary methods were ported correctly.
  1. All calls to `open()` files should be in binary mode.
  1. Unavoidable incompatibilities between Twisted APIs on Python 3 and the original behavior on Python 2 are documented in [source:trunk/doc/core/howto/python3.xhtml doc/core/howto/python3.xhtml] (incompatibilities in the Python 2 behavior are not allowed).
  1. Python 3 changes from Python 2 are addressed, including but not limited to:
    * `map()`, `filter()`, etc. are lazy.
    * `except Exception as e:` for exceptions.
    * Dict `keys()`, `values()` and `items()` are wrapped in `list()` if they are expected to be an unchanging copy.
    * Bytes objects iteration and indexing return integers.
    * No byte paths in `sys.path`.
    * Etc..
  1. `twisted.python.compat` compatibility functions are used, rather than reinventing the wheel.
  1. `_PY3` checks are ideally only done at top of module, e.g. to define alternate helper modules.

== Details ==

  1. Bootstrap the testing system
    1. Port the non-reactor parts of trial necessary to run the tests for the non-reactor parts of trial
      1. [ticket:5853 Refactor the implementation to separate the reactor and non-reactor parts into separate source files]
      1. Add more tests for uncovered functionality
      1. Get the non-reactor parts passing their own tests on [http://www.python.org/download/releases/3.3.0/ Python 3.3]
    1. Test-driven port of the dependencies of the reactor features of trial
    1. Test-driven port of the reactor features of trial
  1. `setup.py` should only install ported modules on Python 3 (#6040, #6096).
  1. Test-driven port of the following modules (some of which may be dependencies of above steps):
    * twisted.application
      * internet
      * service
    * twisted.cred
      * checkers (#5808)
      * credentials (#6176)
    * twisted.trial
    * twisted.protocols
      * basic (#6025)
      * tls (#5979)
    * twisted.internet
      * [ticket:6140 _newtls]
      * [ticket:6141 _sslverify]
      * abstract (#5980, #6163)
      * [ticket:5985 address]
      * [ticket:6065 base]
      * [ticket:5947 defer]
      * endpoints (#6088, #6157)
      * [ticket:5972 error]
      * epollreactor (#6068)
      * [ticket:5974 fdesc]
      * [ticket:6055 gireactor]
      * [ticket:5942 interfaces]
      * pollreactor (#5979, #6068)
      * [ticket:6103 posixbase]
      * [ticket:6060 protocol]
      * reactor
      * selectreactor (#5979, #6068)
      * [ticket:6142 ssl]
      * task (#5979, #6052)
      * [ticket:6067 threads]
      * tcp (#6002, #6069)
      * udp (#6003, #6023)
    * twisted.python
      * [ticket:5983 components]
      * [ticket:6036 deprecate]
      * [ticket:5931 failure] (#5948)
      * [ticket:5932 log]
      * util (#6054, #6158)
      * procutils
      * [ticket:6061 randbytes]
    * twisted.protocols
      * [ticket:6109 loopback]
      * [ticket:6097 policies]
      * [ticket:6139 tls]
    * twisted.names
      * [ticket:6093 cache]
      * [ticket:6056 client]
      * [ticket:6059 common]
      * [ticket:6057 dns]
      * [ticket:6058 error]
      * [ticket:6092 hosts]
    * twisted.web
      * client (#6077)
      * http (#6110)
      * http_headers (#6080, #6082)
      * resource (#6078)
      * static (#6177)
      * server (#6079)
      * util (#6178)

  1. [http://buildbot.twistedmatrix.com/builders/python-3.3-tests Build-slave for Python 3.3 should only run ported test modules.]

== Tips ==

  1. Select a module that needs to be ported (make sure there is a ticket, no one else has started it, etc).
  1. Add the module and its test module(s) to [source:trunk/admin/_twistedpython3.py admin/_twistedpython3.py]
  1. Run [source:trunk/admin/run-python3-tests admin/run-python3-tests]
  1. Fix syntax errors in the test module
    1. If some of the APIs in the module are not being ported, add a `del` statement at the end of the module, guarded by an `if _PY3` check, to get rid of all unported APIs.
  1. If there are syntax errors in modules depended on by the test module:
    1. See if the depended-on module has been split into a `foo` and `_foopy3` module and if the APIs needed from it are already in the `_foopy3` module. 
      1. If so, switch the test module to using the `_foopy3` module directly.
      1. If not, go back to the beginning using the depended-on module as the thing to port.
    1. See if the APIs are really needed.  For example, if the dependency is trial (and trial is not yet ported), perhaps the test module could use (temporarily) the stdlib unittest module instead.
      1. If the APIs are not needed, switch them to something that has been ported or remove their use.
      1. If they are, go back to the beginning using the depended-on module as the thing to port.
  1. Make the tests pass on Python 2.6 and Python 2.7.
  1. Make the tests pass on Python 3.3.
    1. If only some of the APIs in the module are being ported, use the same `del` solution at the end of the test module to get rid of `TestCase` subclasses for unported APIs.
  1. Verify the tests still pass on all 3 versions of Python.
  1. Submit for review

  * Strings
    * Types
      * Python 2 largely conflates text and bytes in the `str` type, but offers `unicode` for unambiguous text usage.
      * Python 3 reduces the functionality of the `str` type and renames it to `bytes`, reflecting the intent that it be used for bytes only.  It renames the `unicode` type to `str`.
      * There is a third kind of string, used to represent Python identifiers and docstrings.  These are bytes in Python 2 and text in Python 3.
    * Plan
      * Python 2.x native (byte) strings will be changed to Python 3.x native (text) strings except where some particular consideration dictates something else.  For example, strings which belong to APIs primarily related to network traffic (ie, strings that come out of or go into sockets) will remain as byte strings.
      * As a consequence, APIs which previously only supported Python 2.x native (byte) strings will be changed on Python 3.x to accept native (text) strings.  Where there does not introduce a new requirement to choose an encoding, arbitrary text will be supported.  Where it does and the choice of encoding is trivially obvious (for example, dealing with a protocol which mandates UTF-8 in an RFC), arbitrary text will be supported.  Where the choice if encoding is not trivially obvious, only ASCII text will be supported.
      * A later porting pass will revisit these ASCII only sites and expand support to include arbitrary text.

  * Running `trial --version` requires the following modules:
   * twisted
   * twisted.application
   * twisted.application.app
   * twisted.application.reactors
   * twisted.application.service
   * [ticket:6040 twisted.copyright]
   * twisted.internet
   * [ticket:5947 twisted.internet.defer]
   * [ticket:6060 twisted.internet.protocol]
   * twisted.internet.utils
   * twisted.persisted
   * twisted.persisted.sob
   * twisted.persisted.styles
   * twisted.plugin
   * twisted.python
   * [ticket:5877 twisted.python.compat] (#6099)
   * twisted.python.components
   * [ticket:5934 twisted.python.context]
   * twisted.python.deprecate (#5916, #6036)
   * [ticket:5931 twisted.python.failure]
   * [ticket:5903 twisted.python.filepath]
   * twisted.python.hashlib
   * [ticket:5860 twisted.python.hook]
   * [ticket:4920 twisted.python._initgroups]
   * [ticket:5960 twisted.python.lockfile]
   * [ticket:5932 twisted.python.log]
   * twisted.python.logfile
   * twisted.python.modules (#6041)
   * [ticket:5896 twisted.python.monkey]
   * [ticket:6061 twisted.python.randbytes]
   * [ticket:5897 twisted.python.runtime]
   * twisted.python.reflect (#5906, #5913, #5914, #6100)
   * twisted.python.text
   * [ticket:5860 twisted.python.threadable]
   * twisted.python.usage
   * twisted.python.util (#5933, #6054)
   * [ticket:5917 twisted.python.versions]
   * [ticket:6039 twisted.python.win32]
   * twisted.python.zippath
   * [ticket:5966 twisted.scripts.trial]
   * twisted.trial
   * twisted.trial.itrial
   * [ticket:5964 twisted.trial.reporter]
   * [ticket:5965 twisted.trial.runner]
   * twisted.trial.unittest (#5874, #5963, #6033)
   * twisted.trial.util (#6019)
   * [ticket:5886 twisted._version]

  Ideally we could omit a number of them:
    * `twisted.python.hook` (#5860)
    * `twisted.python._initgroups` (#4920)
    * ~~`twisted.python.deprecate` - we will not be porting deprecated code, presumably, so hopefully we can omit the module.~~ (trial depends on this, porting it is easier than selectively cutting out all the bits in trial and throughout the test suite that rely on the functionality)
    * `twisted.python.win32` - windows is not a target platform for the initial release.
    * `twisted.persisted.sob` and `twisted.persisted.styles` - this is all useless and/or terrible.
    * No doubt some others we can get away with only porting part of the module.

== Lessons Learned ==

These are things that won't necessarily have any bearing on the Twisted porting project.  However, they may inform future porting efforts, either of Twisted-using applications or any other Python applications.

  * Initial porting effort was estimated by looking at what modules would be needed.  This overlooked some modules because they are only imported inside functions.
  * Splitting up modules into two separate implementation modules, one for ported code and one for unported code, is disruptive (#6183).  Particularly, it's a great source of huge merge conflicts for development not related to the Python 3 port.

== Other ==

Some further ideas to consider: https://wiki.ubuntu.com/Python/3

=== Documentation ===

Documentation potentially requires porting as well.  An individual example (ie, one of the runnable .py files in a ''doc/<project>/examples/'' directory) can be treated much as a module for the purposes of the porting workflow.  One difference, however, is that the example is not added to the list of ported modules in `admin/_twistedpython3.py`.

As with the rest of Twisted, the goal for porting documentation should be a version which is compatible with all supported versions of Python (eg, at this time, Python 2.6, Python 2.7, and Python 3.3).