Support Python 2.6, Python 2.7, and Python 3.3 in a single codebase.

[itamarst]

This is meta-bug covering the work we'll need to do. The goal is having the same source code work in both (Python 2.6 is similar enough to Python 3.3 for this to work).

For each incompatibility between the two, we will open a ticket with keyword py3k, covering the transition. If the transition can be eased by changes to the coding standard, that bit should be split into a separate ticket so that we can apply the ticket ASAP and at least new code will be better.

Compatibility code intended for this process will go in twisted.python.compat. Changes which application developers targetting Python 3 need to be aware of will go in doc/core/howto/python3.xhtml.

Installation documentation also needs to be updated to reflect the possibility of installing on Python 3, as well as the differing requirements that exist when that is done (note #5935).

This may eventually be moved into separate project, since it will be helpful to other project as well.

[itamarst]

This may end up being more like 2.7/3.1 or something, depending how well the transition path happens. The hope is no one will use 3.0.

[thijs]

This ticket doesn't have a milestone yet but putting it in the Python-2.6 milestone so it might get noticed.

[thijs]

Replying to thijs:

This ticket doesn't have a milestone yet but putting it in the Python-2.6 milestone so it might get noticed.

Opened #3432 for a py3k warning mode buildslave that should point out the most obvious differences between 2.6 and 3.0.

[therve]

If it needs a milestone, it would be 3.0 one.

[glyph]

Well,  it's out now.

exarkun posted some results of running the current Python 2.6 version of 2to3 over Twisted:

RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: ws_comma
Traceback (most recent call last):
  File "/home/exarkun/Projects/python/branches/release26-maint/Tools/scripts/2to3", line 6, in <module>
    sys.exit(main("lib2to3.fixes"))
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/main.py", line 92, in main
    rt.refactor(args, options.write, options.doctests_only)
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/refactor.py", line 196, in refactor
    self.refactor_dir(dir_or_file, write, doctests_only)
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/refactor.py", line 214, in refactor_dir
    self.refactor_file(fullname, write, doctests_only)
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/refactor.py", line 237, in refactor_file
    tree = self.refactor_string(input, filename)
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/refactor.py", line 262, in refactor_string
    self.refactor_tree(tree, name)
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/refactor.py", line 301, in refactor_tree
    self.traverse_by(self.post_order, tree.post_order())
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/refactor.py", line 325, in traverse_by
    new = fixer.transform(node, results)
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/fixes/fix_metaclass.py", line 148, in transform
    if not has_metaclass(node):
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/fixes/fix_metaclass.py", line 34, in has_metaclass
    return has_metaclass(node)
  File "/home/exarkun/Projects/python/branches/release26-maint/Lib/lib2to3/fixes/fix_metaclass.py", line 39, in has_metaclass
    if leaf_node.value == '__metaclass__':
AttributeError: 'Node' object has no attribute 'value'

This ticket will definitely need to wait until at least the basic tools work, and z.i has made a similar transition.

Also, I don't think that anyone on the dev team has a real need for 3.0 support; presumably some folks from the community will arrive to take care of this eventually. At any rate, I'm definitely not going to be doing anything about this :).

[glyph]

I noticed #3562 has been opened for a py3 buildslave.

[exarkun]

Apparently the version of 2to3 in Python 2.6 isn't the latest. The traceback above is caused by a bug that's fixed in the latest 2to3.

[glyph]

Where does one get the "latest" 2to3 then? 2.6.1? 3.0?

[exarkun]

 Python svn sandbox

[exarkun]

For anyone interested in contributing to resolving this ticket, fixing test failures on the python-3k-warnings builder is the first step. Builds for that builder are linked from < http://buildbot.twistedmatrix.com/builders/python-3k-warnings>; problems are linked from individual build pages.

[Screwtape]

For the record, the test failures on the python-3k-warnings builder are a very sketchy and incomplete record of what needs to be fixed - nearly every test prints a bunch of warnings that aren't noticed by the test suite, and Trial spews a bunch before it even gets to the first test.

Here's a command line that gives you a good idea of what things need to be cleaned up for optimum Python 3.x compatibility:

python2.6 -3 bin/trial --reporter=summary twisted 2>&1 |
    grep "$PWD.*DeprecationWarning" |
    cut -d" " -f3- |
    egrep -iv 'twisted|reactor' |
    sort | uniq -c | sort -n

It tries to filter out the various warnings issued by Twisted and the Python standard library (fun fact: the 2.6 stdlib is not Python 3.x-safe), and sort the resulting messages by increasing frequency. Currently, it produces the following output:

      1 catching classes that don't inherit from BaseException is not allowed in 3.x
      1 the cPickle module has been removed in Python 3.0
      1 The 'hotshot' module is not supported in 3.x, use the 'profile' module instead.
      1 the ihooks module has been removed in Python 3.0
      1 the MimeWriter module is deprecated; use the email package instead
      1 The popen2 module is deprecated.  Use the subprocess module.
      2 dict inequality comparisons not supported in 3.x
      4 Overriding __cmp__ blocks inheritance of __hash__ in 3.x
      6 apply() not supported in 3.x; use func(*args, **kwargs)
      8 BaseException.message has been deprecated as of Python 2.6
     11 Overriding __eq__ blocks inheritance of __hash__ in 3.x
     15 execfile() not supported in 3.x; use exec()
     20 In 3.x, reload() is renamed to imp.reload()
     21 exceptions must derive from BaseException in 3.x
     29 map(None, ...) not supported in 3.x; use list(...)
     44 __getitem__ not supported for exception classes in 3.x; use args attribute
     65 comparing unequal types not supported in 3.x
     66 reduce() not supported in 3.x; use functools.reduce()
    105 classic int division
    202 the cmp argument is not supported in 3.x
    847 builtin_function_or_method inequality comparisons not supported in 3.x
    856 buffer() not supported in 3.x
   1970 dict.has_key() not supported in 3.x; use the in operator
  16702 callable() not supported in 3.x; use hasattr(o, '__call__')

Some of these are simple, mechanical substitutions, some of them have a few corner-cases (there are parts where a dictionary's .has_key() method is passed around or stored, resulting in warnings seemingly arising on lines which don't call .has_key()), some are compatibility gotchas (list.sort(cmp=foo) is not supported after 3.0, list.sort(key=foo) in not supported before 2.4 but Twisted still tries to be compatible with 2.3), and some are fiendishly complicated (the 'X blocks inheritance of hash' needs some heavy thought for each case, I expect).

Pick a warning, file a bug and have at it, I guess.

[Screwtape]

I've filed ticket #4053 about "dict.has_key() not supported in 3.x; use the in operator".

[cvaliente]

Ticket #4065 takes care of "`callable() not supported in 3.x; use hasattr(o, 'call') `"

[cvaliente]

Ticket #4066 takes care of "reduce() not supported in 3.x; use functools.reduce()"

[loewis]

I have created one ticket that allows running setup.py on 3.x (#4244), and four tickets that deal with porting C modules (#4245, #4246, #4247, #4248). Combined, they allow 'setup.py build' to complete on 3.x.

[glyph]

#4764 was a duplicate of this.

[glyph]

Replying to loewis:

I have created one ticket that allows running setup.py on 3.x (#4244), and four tickets that deal with porting C modules (#4245, #4246, #4247, #4248). Combined, they allow 'setup.py build' to complete on 3.x.

Sorry that these

[glyph]

Replying to glyph:

Replying to loewis:

I have created one ticket that allows running setup.py on 3.x (#4244), and four tickets that deal with porting C modules (#4245, #4246, #4247, #4248). Combined, they allow 'setup.py build' to complete on 3.x.

Sorry that these

Ahem. Thank you trac. loewis: sorry that some of these have languished for a while, but I just wanted to note that I appreciate that you broke up the patches into these manageable chunks.

[thijs]

updating description with py3k keyword reference.

[thijs]

Replying to Screwtape:

Here's a command line that gives you a good idea of what things need to be cleaned up for optimum Python 3.x compatibility:

Thanks, that works great, just takes a long time :) Current trunk (r32047) reports:

1 <> not supported in 3.x; use !=
   1 In 3.x, os.path.walk is removed in favor of os.walk.
   1 The 'hotshot' module is not supported in 3.x, use the 'profile' module instead.
   1 catching classes that don't inherit from BaseException is not allowed in 3.x
   1 the commands module has been removed in Python 3.0; use the subprocess module instead
   1 the ihooks module has been removed in Python 3.0
   2 classic long division
   2 dict inequality comparisons not supported in 3.x
   6 In 3.x, reload() is renamed to imp.reload()
   9 Overriding __eq__ blocks inheritance of __hash__ in 3.x
   9 apply() not supported in 3.x; use func(*args, **kwargs)
  12 __getitem__ not supported for exception classes in 3.x; use args attribute
  18 exceptions must derive from BaseException in 3.x
  18 execfile() not supported in 3.x; use exec()
  51 comparing unequal types not supported in 3.x
 111 classic int division
 263 the cmp argument is not supported in 3.x
 562 buffer() not supported in 3.x
 682 dict.has_key() not supported in 3.x; use the in operator
17900 callable() not supported in 3.x; use isinstance(x, collections.Callable)

I've opened tickets for a couple of the first warnings: #5136, #5137, #5138

[loewis]

So are solutions involving 2to3 acceptable or not?

[itamar]

We're changing the code so it supports 2.6, 2.7 and 3.3 off the same code base. Getting to that point using 2to3 is fine, but there's always a need for human judgement to make sure it did the right thing. For example, keeping string literals as native strings, i.e. Unicode on Python 3, is often the wrong thing to do. Moreover ensuring test coverage so we know result works is also needed.

Also we've written some compatibility stuff which 2to3 wouldn't know about, e.g. a class decorator that allows use of __cmp__ in Python 3. See twisted.python.compat in trunk for what we've got so far.

[exarkun]

Getting to that point using 2to3 is fine, but there's always a need for human judgement to make sure it did the right thing.

Personally, I think 2to3 is a waste of time. Verifying that its output is correct is at least as much work as making the correct changes manually. I'd strongly prefer not to see any patches generated by 2to3 submitted for review.

[loewis]

2to3 would provide the advantage of also supporting 2.5 and 2.4 "easily". In that approach, 2to3 output would never be manually inspected (except when fixing bugs); instead, the source that is edited is always the 2.x source, and 2to3 runs whenever "setup.py build" runs.

For example, this would allow continued use of "except Type,value:". "setup.py build" would convert this into "except Type as value" when run under 3.x, but leave it unmodified when running under 2.x. In a single code base, you would have to write "except Type as value" into the original source, causing it to break on 2.5. The work-around for getting at the exception object in a single source is really ugly and involves calling sys.exc_info().

If a definite decision is made to not support 2.5 anymore, then using 2to3 is of less value. The only major concern is then Unicode literals, which need to be spelled as u("text") if 3.2 needs to be supported. In general, the approach needs a lot of helper functions which got integrated in the six package, which might then be useful to integrate into Twisted.

[exarkun]

Neither Python 2.5 nor Python 3.2 will be supported.