[Twisted-Python] Twisted for Python 3

[Glyph]

On Oct 19, 2011, at 4:28 PM, Itamar Turner-Trauring wrote:

>> To me, 3.x support in trunk is not really workable unless you drop
>> pre-2.6 compatibility. Bytes literals are pretty much required.
>> A possible workaround is to call a factory function that makes
>> an str into a bytes in 3.x (e.g. write B("xyz") everywhere instead of
>> b"xyz"), but that's quite ugly IMO, not to mention suboptimal.
> 
> We're getting closer; we're going to drop Python 2.4 as soon as the next
> release is out. Still not up to 2.6, though.

For one thing:

What's the big problem with B("")?  It does not strike me as particularly problematic.  We call plenty of functions at import time, I'm sure that it's not going to impact performance much.  Plus, it's very easy to fix with a trivial regex when we do drop 2.5 support eventually: the expression evaluates to the same thing.  If anything, concern about the function call overhead will simply encourage us to make more constant bytestrings module-level constants instead of inline expressions, which is a plus if you ask me :).

>> Incompatibilities are a given when you switch between two different
>> data models. The incompatibilities don't have to be wire-level, but
>> they *will* occur at the higher level anyway.
> 
> The thing that worries is me is unnecessary or harmful incompatibilities
> that are a result of misunderstanding (e.g. the banana thing, which was an
> unnecessary wire protocol change). Submitting patches would have the
> benefit of letting someone else help you make these decisions; it's
> unreasonable to expect you to become an expert on every single API in
> Twisted.

Nothing to say to this besides a big "+1".  This is the whole point of the review process: to make sure that appropriate knowledge is shared before committing to a change.

>> Without wanting to sound unconstructive, the "right way" (the
>> "byte-sized" approach) does not sound workable. Some 3.x-related patches
>> (not mine) have been lingering for years, although they are trivial. And
>> these patches (mine and the others) are really the tip of iceberg.
> 
> There aren't any python 3 patches in the review queue at the moment; do
> you have any specific tickets in mind? If they're ready to go they should
> have the "review" keyword set (if they haven't addressed review comments
> then that is why they aren't merged).

I should note also that our review queue has gotten to 0 tickets several times since those tickets have last been in review, so things are getting through.

The problem here is that nobody is putting in the consistent work necessary to get these patches through review and merged to trunk.  Almost all the py3k patches have been dropped off and then abandoned.  The solution is not to give up and abandon all the patches together, but rather to find some people to get involved and continue participating in that work.

(That's not to say that I don't appreciate the effort involved in dropping those patches off in the first place.  I do.  It also doesn't mean that people shouldn't drop off patches if they're not going to finish them; they definitely should.  The original author doesn't necessarily have to be the one to do the responding to feedback.)

It may seem easier right now to put them into a big pile and forget about that process, but what happens once trunk has moved on?  We don't make a lot of incompatible changes, but that doesn't mean we don't change the implementation of things quite a bit.  Lots of work has gone on in core areas of Twisted recently - including a near-total rewrite of TLS support - and many more such changes are coming.  The relative stability of Twisted as an API to depend on says nothing about the stability of the code in terms of diffs continuing to apply; and the larger the diff, the more likely it's going to run into conflicts.

It's worth the extra effort to get the code into the mainline because when the work is done, it's actually done: we continue to maintain that code moving forward, and assuming that there is an appropriate buildslave (possibly one with the 'rachet'-style reporting that exarkun just set up for pydoctor?), we make sure that future changes won't undo the progress that has been made and introduce more py3k warnings or test failures.

If the code isn't in the mainline, then instead of supporting your efforts, all future maintenance undermines them.  You'll have to put in lots of extra work to keep your fork applying, and you'll have to do a bunch of work because any changes will not take the py3 compatibility tests into account, so every new module will be a new thing you have to clean up.  Plus you don't get the benefit of having each change tested extensively to make sure it doesn't break anything unexpected, on some weird platform or configuration.

This work on the build infrastructure and release management and QA appears to be invisible from the perspective of an individual change, but in reality it's the plurality, if not the majority, of the work that goes into the project.

>> *Assuming* we find a clean solution to the bytes literal problem,

We've already got one, except perhaps for an unreasonably strict definition of "clean" :-).

>> I could try to slice all of that into "byte-sized" patches into which I
>> inject version-checking boilerplate, but that would mean a large waste
>> of time and energy:
>> - for me, as I post dozens of small patches and have to follow up on
>>  them, and wait for them to be checked in

You can choose the appropriate size of patches; you don't have to work on a per-file basis; it would make more sense to choose a topic area than to just submit individual one-liners as patches because that's what happens to be in a single file.

>> - for you, as you have to review these patches without perhaps even
>>  being Python 3 users yourselves, and without Python 3 compatibility
>>  being on your priority list
> 
> Most of us aren't Windows users either, but we still try to support
> Windows :)

Also, the issue is not the priority of python 3 support, but the priority of reviewing incoming patches, which is always at the absolute top of our priority list.  As you can see from the scoring system on <http://twistedmatrix.com/highscores/>, we value it more highly than anything else.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20111019/1f0fb42c/attachment-0001.htm