wiki:Plan/Python3

Version 21 (modified by exarkun, 2 years ago) (diff)

Consistently expose hierarchy

Objective

To have Twisted available on Python 3.3 and newer, with the same functionality as is currently available on Python 2.x.

Strategy

  • Single source - The result will be a version of Twisted that is source-compatible with Python 2.6, Python 2.7, and Python 3.3.
  • Test-driven - Following the standard Twisted development process, all changed code will be fully automatically unit tested.
  • Opportunistic - Where module dependencies (within Twisted) need to be ported to support porting of the high-priority modules, modules may be split so that only some of their contents need to be ported.

All other standard Twisted development practices apply.

An initial minimal project (http://twistedmatrix.com/trac/milestone/Python%203.3%20Minimal) is being funded by Canonical and implemented by Future Foundries LLC (i.e. Itamar and Jean-Paul), with the aim of getting a basic subset of Twisted working, sufficient to run a specific Canonical project. If that is done and any time is left over in the contract, it will be spent doing more porting, so help by other developers will hopefully result in more of Twisted being ported.

Reviewer check list

  1. Tests pass under 3.3.
  2. Ported modules were added to list of ported modules - admin/_twistedpython3.py
  3. No warnings when running using -3 mode under Python 2.7.
  4. All non-docstring strings are clearly marked as one of the various string categories (bytes/unicode/identifier/version-dependent).
  5. No use of 'str' except for things that really can differ across versions of Python.
  6. from __future__ import division, absolute_import was added to the module.
    1. And any code changes necessary to work with the resulting changed semantics were made.
  7. Verify dictionary methods were ported correctly.
  8. All calls to open() files should be in binary mode.

Details

  1. Bootstrap the testing system
    1. Port the non-reactor parts of trial necessary to run the tests for the non-reactor parts of trial
      1. Refactor the implementation to separate the reactor and non-reactor parts into separate source files
      2. Add more tests for uncovered functionality
      3. Get the non-reactor parts passing their own tests on Python 3.3 (alpha, until 3.3 is actually released)
    2. Test-driven port of the dependencies of the reactor features of trial
    3. Test-driven port of the reactor features of trial
  2. setup.py should only install ported modules on Python 3.
  3. Test-driven port of the following modules (some of which may be dependencies of above steps):
    • twisted.cred
    • twisted.trial
    • twisted.protocols
      • basic
      • tls
    • twisted.web
      • client
      • http
      • resource
      • server
      • http_headers
    • twisted.internet
      • defer
      • gireactor
      • selectreactor
      • pollreactor
      • ssl
      • abstract
      • reactor
      • endpoints
      • protocol
      • interfaces
      • error
      • task
    • twisted.python
      • failure
      • log
      • util
      • procutils
    • twisted.names
    • twisted.application
      • internet
      • service
  4. Build-slave for Python 3.3 should only run ported test modules.

Tips

  1. Select a module that needs to be ported (make sure there is a ticket, no one else has started it, etc).
  2. Add the module and its test module(s) to admin/_twistedpython3.py
  3. Run admin/run-python3-tests
  4. Fix syntax errors in the test module
    1. If only some of the APIs in the module are being ported, split the test module into two. Move the tests for the APIs being ported into a new module, test_foopy3.py. For the remaining steps, just fix the test_foopy3.py module.
  5. If there are syntax errors in modules depended on by the test module:
    1. See if the depended-on module has been split into a foo and _foopy3 module and if the APIs needed from it are already in the _foopy3 module.
      1. If so, switch the test module to using the _foopy3 module directly.
      2. If not, go back to the beginning using the depended-on module as the thing to port.
    2. See if the APIs are really needed. For example, if the dependency is trial (and trial is not yet ported), perhaps the test module could use (temporarily) the stdlib unittest module instead.
      1. If the APIs are not needed, switch them to something that has been ported or remove their use.
      2. If they are, go back to the beginning using the depended-on module as the thing to port.
  6. Make the tests pass on Python 2.6 and Python 2.7.
  7. Make the tests pass on Python 3.3.
    1. If only some of the APIs in the module are being ported, split the implementation module into two. Move the implementation of the APIs being ported into a new file, _foopy3.py. Import these names into the original module to preserve the API. Import the APIs directly from this new private module in the test module, so the implementation of the APIs not being ported don't get in the way of running the tests.
  8. Verify the tests still pass on all 3 versions of Python.
  9. Submit for review
  • Strings
    • Types
      • Python 2 largely conflates text and bytes in the str type, but offers unicode for unambiguous text usage.
      • Python 3 reduces the functionality of the str type and renames it to bytes, reflecting the intent that it be used for bytes only. It renames the unicode type to str.
      • There is a third kind of string, used to represent Python identifiers and docstrings. These are bytes in Python 2 and text in Python 3.
    • Plan
      • Python 2.x native (byte) strings will be changed to Python 3.x native (text) strings except where some particular consideration dictates something else. For example, strings which belong to APIs primarily related to network traffic (ie, strings that come out of or go into sockets) will remain as byte strings.
      • As a consequence, APIs which previously only supported Python 2.x native (byte) strings will be changed on Python 3.x to accept native (text) strings. Where there does not introduce a new requirement to choose an encoding, arbitrary text will be supported. Where it does and the choice of encoding is trivially obvious (for example, dealing with a protocol which mandates UTF-8 in an RFC), arbitrary text will be supported. Where the choice if encoding is not trivially obvious, only ASCII text will be supported.
      • A later porting pass will revisit these ASCII only sites and expand support to include arbitrary text.
  • Running trial --version requires the following modules:
    • twisted
    • twisted.application
    • twisted.application.app
    • twisted.application.reactors
    • twisted.application.service
    • twisted.copyright
    • twisted.internet
    • twisted.internet.defer
    • twisted.internet.protocol
    • twisted.internet.utils
    • twisted.persisted
    • twisted.persisted.sob
    • twisted.persisted.styles
    • twisted.plugin
    • twisted.python
    • twisted.python.compat
    • twisted.python.components
    • twisted.python.context
    • twisted.python.deprecate
    • twisted.python.failure
    • twisted.python.filepath
    • twisted.python.hashlib
    • twisted.python.hook
    • twisted.python._initgroups
    • twisted.python.lockfile
    • twisted.python.log
    • twisted.python.logfile
    • twisted.python.modules
    • twisted.python.monkey
    • twisted.python.runtime
    • twisted.python.text
    • twisted.python.threadable
    • twisted.python.usage
    • twisted.python.util
    • twisted.python.versions
    • twisted.python.win32
    • twisted.python.zippath
    • twisted.scripts.trial
    • twisted.trial
    • twisted.trial.itrial
    • twisted.trial.reporter
    • twisted.trial.runner
    • twisted.trial.unittest
    • twisted._version

Ideally we could omit a number of them:

  • `twisted.python.hook (#5860)
  • twisted.python._initgroups (#4920)
  • twisted.python.deprecate - we will not be porting deprecated code, presumably, so hopefully we can omit the module. (trial depends on this, porting it is easier than selectively cutting out all the bits in trial and throughout the test suite that rely on the functionality)
  • twisted.python.win32 - windows is not a target platform for the initial release.
  • twisted.persisted.sob and twisted.persisted.styles - this is all useless and/or terrible.
  • No doubt some others we can get away with only porting part of the module.