[Twisted-Python] Tubes!

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Thu Jul 10 13:32:53 MDT 2014


Hello all,

Some of you may have heard rumors of some work in progress on a 
replacement for Twisted's IConsumer/IProducer interfaces.

Tubes have been largely Glyph's effort (though a lot of people have 
contributed in one way or another).  And a large effort it's been. 
Development is proceeding in a Twisted branch and comes to over three 
thousand lines of additions so far.

Given the large size of the implementation and the long time that this 
effort has been underway (I remember the Twisted meetup at the Rackspace 
offices that *I* attended when I was visiting SF... a year and a half 
ago... at which point tubes wasn't exactly a brand new project) I'd like 
to re-raise the idea that the best next step for the project is to see 
some distribution in its *current* state.

Specifically, I think it would be beneficial to set up a tubes project 
on Github under the Twisted organization and try for a release in the 
very near future.

I think this has several advantages over the status quo:

  1) As an independent project, tubes will attract more attention than it 
presently gets as a relatively unknown ticket & branch of Twisted.

  2) As a separate Python package, the logistics of actually using tubes 
are simpler (just consider how you might declare a dependency on a 
branch of Twisted - keeping in mind you may want to use tubes in a 
project that already depends on some version of Twisted).  It may not 
make sense to say that it is the same quality as Twisted proper right 
off the bat (on the other hand, it may well - I suspect tubes in its 
current form actually is a lot higher quality than large sections of 
Twisted) but that doesn't mean people (not to mention the tubes project) 
can't benefit from being able to experiment with it.

  3) Decoupling tubes from Twisted frees tubes from certain of Twisted's 
policies which are more challenging to follow for the kind of non- 
trivial, brand new code base that tubes is.  Technically we could just 
say that these policies don't apply to a tubes package *in* Twisted but 
this kind of subtle distinction is often lost on users (ie application 
developers).

    a) Twisted's compatibility policy need not apply.  It could either be 
sped up or abandoned more thoroughly.  I'm generally a fan of being 
backwards compatible even when you have few users because it actually 
makes development easier, but loosening the policy to say things might 
break if it's just really inconvenient to keep them working (whereas 
Twisted goes to the inconvenience to keep them working) seems 
reasonable.

    b) tubes can undergo a faster release cycle to benefit more from user 
feedback.

  4) At this point, a normal review of the tubes branch is going to be a 
problem.  We do not have good tools or mechanisms for dealing with 
branches this large.  The code in the current tubes branch can just 
become master of a new project.  Development going forward from this 
point should continue to follow the feature-branch, small-changes, pre- 
commit-peer-review process.  But those 3k lines are written already. 
Short of an extremely expensive effort to break the work up into 
smaller, self-contained pieces there's simply never going to be a *good* 
review in the typical style.

Additionally, it may turn out that tubes can remain independent 
indefinitely.  Someday perhaps Twisted would come to depend on it to 
allow the various protocols and applications implemented in Twisted to 
benefit from the superior abstractions it provides.  Or maybe once it 
has undergone a few iterations it will make sense to bring it back to 
Twisted.  I don't think this needs to be decided now.

There are downsides, of course.  All of the boring maintenance involved 
with having a separate project - setting up CI, actually doing the 
releases, etc.  Perhaps we could find some volunteers to help out with 
these tasks, though, in exchange for getting some great code out there?

I'm curious what the folks out there who develop applications using 
Twisted would find to be the easiest path forward.  I'm also curious to 
hear what Glyph thinks about all this. ;)

Jean-Paul



More information about the Twisted-Python mailing list