[Twisted-Python] Twisted for Python 3

Antoine Pitrou solipsis at pitrou.net
Wed Oct 19 13:58:36 MDT 2011


Hi Glyph,

On Wed, 19 Oct 2011 14:50:41 -0400
Glyph <glyph at twistedmatrix.com> wrote:
> On Oct 19, 2011, at 7:50 AM, Antoine Pitrou wrote:
> 
> > Feedback and contributions welcome. If you are interested in helping,
> > please go and read the link above, it will give you suggestions.
> 
> We do eventually want to have 3.x support in trunk, so anything you're doing to support that goal should be an acceptable patch - provided it meets our other quality criteria, of course, but there's no patch for py3k support which shouldn't be able to meet that criteria :).

To me, 3.x support in trunk is not really workable unless you drop
pre-2.6 compatibility. Bytes literals are pretty much required.
A possible workaround is to call a factory function that makes
an str into a bytes in 3.x (e.g. write B("xyz") everywhere instead of
b"xyz"), but that's quite ugly IMO, not to mention suboptimal.

> I would suggest that you submit all patches directly to the Twisted tracker, where they can be properly reviewed before landing, not to this (or any other) fork.  As several other messages in this thread have indicated, it looks like you may be misunderstanding a few key aspects of Twisted (such as how the distinction between bytes and unicode needs to be treated in the Banana protocol), so this fork is going to get some things wrong, potentially with wire-level implications and incompatibilities.

Incompatibilities are a given when you switch between two different
data models. The incompatibilities don't have to be wire-level, but
they *will* occur at the higher level anyway.

You may read my answers to Itamar for more explanation on that decision.
Of course, it is not set in stone.

> Most importantly, it will not be reasonable to merge in a humongous branch with hundreds of unrelated changes later; doing code review on changes that big just isn't feasible and chances are good that they will linger forever.  If your fork _is_ made to work correctly, folding that correct behavior back in with all the necessary test coverage and fixes will be a much larger job than doing it the right way in the first place, one bit at a time, in Twisted trunk.

Without wanting to sound unconstructive, the "right way" (the
"byte-sized" approach) does not sound workable. Some 3.x-related patches
(not mine) have been lingering for years, although they are trivial. And
these patches (mine and the others) are really the tip of iceberg.

To clarify, here's my current set of modifications post-2to3 (that
is, it does *not* include 2to3's own changes):

 .hgignore                                  |    7 +
 bin/manhole                                |    2 +-
 bin/mktap                                  |    2 +-
 bin/pyhtmlizer                             |    2 +-
 bin/tap2deb                                |    2 +-
 bin/tap2rpm                                |    2 +-
 bin/tapconvert                             |    2 +-
 bin/trial                                  |    2 +-
 bin/twistd                                 |    2 +-
 doc/core/examples/chatserver.py            |    2 +-
 doc/core/examples/echoclient.py            |    6 +-
 doc/core/examples/echoclient_ssl.py        |    6 +-
 doc/core/examples/echoclient_udp.py        |    6 +-
 doc/core/examples/simpleclient.py          |    2 +-
 setup.py                                   |    6 +-
 twisted/application/internet.py            |    4 +-
 twisted/enterprise/adbapi.py               |    8 +
 twisted/internet/_sigchld.c                |   15 +-
 twisted/internet/_signals.py               |    2 +-
 twisted/internet/_sslverify.py             |    9 +-
 twisted/internet/abstract.py               |   10 +-
 twisted/internet/address.py                |    9 +-
 twisted/internet/base.py                   |   14 +-
 twisted/internet/defer.py                  |   13 +
 twisted/internet/error.py                  |    4 +-
 twisted/internet/inotify.py                |    4 +-
 twisted/internet/pollreactor.py            |    2 +-
 twisted/internet/posixbase.py              |    4 +-
 twisted/internet/process.py                |    8 +-
 twisted/internet/protocol.py               |    2 +-
 twisted/internet/task.py                   |    2 +-
 twisted/internet/tcp.py                    |    7 +-
 twisted/internet/test/connectionmixins.py  |   10 +-
 twisted/internet/test/fakeendpoint.py      |    5 +-
 twisted/internet/test/test_fdset.py        |   18 +-
 twisted/internet/test/test_inotify.py      |    6 +-
 twisted/internet/test/test_posixprocess.py |    2 +-
 twisted/internet/test/test_process.py      |   33 +-
 twisted/internet/test/test_tcp.py          |   22 +-
 twisted/internet/test/test_udp.py          |    2 +-
 twisted/internet/test/test_unix.py         |    2 +-
 twisted/internet/threads.py                |    6 +-
 twisted/internet/unix.py                   |    5 +-
 twisted/internet/utils.py                  |   13 +-
 twisted/lore/latex.py                      |    2 +-
 twisted/lore/lint.py                       |    4 +-
 twisted/lore/tree.py                       |   21 +-
 twisted/manhole/explorer.py                |   25 +-
 twisted/persisted/aot.py                   |  146 +++---
 twisted/persisted/crefutil.py              |    2 +-
 twisted/persisted/sob.py                   |    2 +-
 twisted/persisted/styles.py                |    8 +-
 twisted/plugin.py                          |    5 +-
 twisted/protocols/amp.py                   |  210 +++++-----
 twisted/protocols/basic.py                 |   43 +-
 twisted/protocols/finger.py                |   18 +-
 twisted/protocols/ftp.py                   |    4 +-
 twisted/protocols/ident.py                 |   41 +-
 twisted/protocols/loopback.py              |   10 +-
 twisted/protocols/memcache.py              |   47 +-
 twisted/protocols/policies.py              |    4 +-
 twisted/protocols/postfix.py               |   10 +-
 twisted/protocols/socks.py                 |   22 +-
 twisted/protocols/stateful.py              |    7 +-
 twisted/protocols/test/test_tls.py         |   73 +-
 twisted/protocols/tls.py                   |    2 +-
 twisted/protocols/wire.py                  |   10 +-
 twisted/python/_initgroups.c               |   19 +-
 twisted/python/_inotify.py                 |    3 +-
 twisted/python/_release.py                 |   10 +-
 twisted/python/failure.py                  |   25 +-
 twisted/python/filepath.py                 |   44 ++-
 twisted/python/hook.py                     |    4 +-
 twisted/python/htmlizer.py                 |    5 +-
 twisted/python/log.py                      |    4 -
 twisted/python/logfile.py                  |    8 +-
 twisted/python/randbytes.py                |   11 +-
 twisted/python/reflect.py                  |    6 +-
 twisted/python/runtime.py                  |    2 +-
 twisted/python/test/modules_helpers.py     |    2 +-
 twisted/python/test/test_deprecate.py      |    6 +-
 twisted/python/test/test_dist.py           |   10 +-
 twisted/python/test/test_hashlib.py        |   40 +-
 twisted/python/test/test_htmlizer.py       |   18 +-
 twisted/python/test/test_release.py        |  470 ++++++++++++-----------
 twisted/python/test/test_util.py           |   11 +-
 twisted/python/test/test_versions.py       |   19 +-
 twisted/python/test/test_zipstream.py      |  116 +++--
 twisted/python/text.py                     |   24 +-
 twisted/python/usage.py                    |   12 +-
 twisted/python/util.py                     |   74 +---
 twisted/python/versions.py                 |   51 +-
 twisted/python/win32.py                    |    6 +-
 twisted/python/zippath.py                  |   22 +-
 twisted/python/zipstream.py                |   28 +-
 twisted/runner/portmap.c                   |   18 +-
 twisted/scripts/_twistd_unix.py            |    2 +-
 twisted/scripts/trial.py                   |    4 +-
 twisted/spread/banana.py                   |   68 ++-
 twisted/spread/flavors.py                  |   38 +
 twisted/spread/jelly.py                    |  112 +----
 twisted/spread/pb.py                       |   35 +-
 twisted/spread/publish.py                  |    4 +-
 twisted/test/iosim.py                      |    4 +-
 twisted/test/proto_helpers.py              |   10 +-
 twisted/test/test_adbapi.py                |    6 +-
 twisted/test/test_amp.py                   |  330 ++++++++--------
 twisted/test/test_application.py           |   10 +-
 twisted/test/test_banana.py                |   64 +-
 twisted/test/test_defgen.py                |    2 +-
 twisted/test/test_epoll.py                 |    4 +-
 twisted/test/test_failure.py               |    4 +-
 twisted/test/test_fdesc.py                 |   26 +-
 twisted/test/test_finger.py                |   16 +-
 twisted/test/test_ident.py                 |   42 +-
 twisted/test/test_internet.py              |   28 +-
 twisted/test/test_iutils.py                |   18 +-
 twisted/test/test_jelly.py                 |   44 +-
 twisted/test/test_log.py                   |   22 +-
 twisted/test/test_logfile.py               |    6 +-
 twisted/test/test_loopback.py              |   64 +-
 twisted/test/test_memcache.py              |  194 ++++----
 twisted/test/test_modules.py               |   26 +-
 twisted/test/test_newcred.py               |   24 +-
 twisted/test/test_paths.py                 |   70 +-
 twisted/test/test_pb.py                    |   18 +-
 twisted/test/test_pbfailure.py             |    4 +-
 twisted/test/test_persisted.py             |   14 +-
 twisted/test/test_policies.py              |   49 +-
 twisted/test/test_postfix.py               |   20 +-
 twisted/test/test_protocols.py             |  290 +++++++-------
 twisted/test/test_socks.py                 |  112 ++--
 twisted/test/test_ssl.py                   |   28 +-
 twisted/test/test_sslverify.py             |    2 +-
 twisted/test/test_stateful.py              |    8 +-
 twisted/test/test_tcp.py                   |   34 +-
 twisted/test/test_tcp_internals.py         |    5 +
 twisted/test/test_threadable.py            |    2 +-
 twisted/test/test_threads.py               |    6 +-
 twisted/test/test_tpfile.py                |    8 +-
 twisted/test/test_twistd.py                |   14 +-
 twisted/test/test_udp.py                   |   40 +-
 twisted/test/test_unix.py                  |    8 +-
 twisted/trial/runner.py                    |   41 +-
 twisted/trial/test/packages.py             |    2 +-
 twisted/trial/test/test_assertions.py      |    2 -
 twisted/trial/test/test_loader.py          |   21 +-
 twisted/trial/test/test_reporter.py        |   26 +-
 twisted/trial/test/test_tests.py           |   15 +-
 twisted/trial/test/test_util.py            |    1 +
 twisted/trial/test/test_warning.py         |    8 +-
 twisted/trial/unittest.py                  |    3 +
 twisted/trial/util.py                      |    2 +-
 twisted/web/microdom.py                    |    5 +-
 154 files changed, 2171 insertions(+), 1970 deletions(-)


*Assuming* we find a clean solution to the bytes literal problem, I
could try to slice all of that into "byte-sized" patches into which I
inject version-checking boilerplate, but that would mean a large waste
of time and energy:
- for me, as I post dozens of small patches and have to follow up on
  them, and wait for them to be checked in
- for you, as you have to review these patches without perhaps even
  being Python 3 users yourselves, and without Python 3 compatibility
  being on your priority list

It would also mean neither of us would be able to remain motivated
(which, in the end, is the fundamental problem).

Therefore, my work assumes a different approach. I expect 3.x support
for Twisted to need maintenance of a dedicated branch for a
non-ephemeral lapse of time. One goal of my experiment is to find out
how easy or tedious such maintenance is.

> Finally, having the code in mainline will allow it to be subjected to our correctness testing <http://buildbot.twistedmatrix.com/> and also our performance testing <http://speed.twistedmatrix.com/>, neither of which is a small effort.

Setting up buildbots would be very nice indeed. But isn't it independent
from having the code live in "trunk"?
As for performance testing, it's probably prematurate right now :-)

> At any rate, the very first step here should be to add an as-yet-unsupported py3 builder to our build farm.  This should be pretty straightforward, as new builders are added all the time, and we have some new hardware that could be put to use for this.
> 
> Alternatively, we do already have a py3k-warnings buildbot set up; you can see the current warnings here: <http://buildbot.twistedmatrix.com/builders/python-3k-warnings/builds/111/steps/trial/logs/stdio>, although possibly this needs to be updated to python 2.7 to get rid of irrelevant noise like warnings about callable() going away, since I believe it's come back to 3.x now.

As I said, getting rid of py3k warnings is really the tip of the
iceberg. It's a nice first step, but it's neither necessary (if you
plan to maintain a 3.x branch), and it doesn't take you very far away.

(yes, callable() has come back in 3.2)

Regards

Antoine.






More information about the Twisted-Python mailing list