[Twisted-Python] Tubes!

[Glyph]

> On Jul 10, 2014, at 21:32, exarkun at twistedmatrix.com wrote:
> 
> Tubes have been largely Glyph's effort (though a lot of people have contributed in one way or another).  And a large effort it's been. Development is proceeding in a Twisted branch and comes to over three thousand lines of additions so far.
> 
> Given the large size of the implementation and the long time that this effort has been underway (I remember the Twisted meetup at the Rackspace offices that *I* attended when I was visiting SF... a year and a half ago... at which point tubes wasn't exactly a brand new project) I'd like to re-raise the idea that the best next step for the project is to see some distribution in its *current* state.

When Jean-Paul originally wrote this message, I strongly disagreed.

In the intervening six months, I've been working on Tubes on and off.  In the last couple of weeks, I've had a bit of time to make a concerted effort to improve the branch, and it is starting to approach something like a finished state: there is still a lot of polish necessary, but as I've tested them against more and more use-cases, the core abstractions seem to be holding up rather well, and several rounds of increasingly elaborate use-cases did not result in any big rewrites or mystifying un-debuggable re-entrancy scenarios in the implementation.

One change Thomas Hervé assisted me in making today was to remove Deferred support from the core of "tube" itself, and instead factor it out into a separate module.  Interested readers may find that module here:

<https://github.com/twisted/twisted/blob/tubes-1956-8/twisted/tubes/undefer.py>

It's not finished (mostly it needs testing for errbacks and the addition of yielding non-Deferred values).

This change radically simplifies the conceptual model around consuming asynchronous results within the implementation. If you want to deal with results instead of Deferreds, you just create this tube that turns Deferreds into their results - the only special-casing inside tube itself is a special value that says "when you yield this, don't actually emit an output to the next drain in the series".

Another thing that this change does is illustrate the fact that literally nothing in the core part of Tubes has come to depend on Twisted - and indeed, as the factoring has improved over time, the coupling between Twisted and the core Tubes abstraction has become looser and looser.  This is a very good thing architecturally.  Examining the few remaining points of coupling, there are pretty much only two: twisted's logging serves as the global error-handler hook (which can be easily changed to a simple bootstrapping API) and the use of Failure as the "reason" parameter to the .flowStopped and .stopped methods.  This was copied from IProtocol, of course, but I think that upon reflection it is a bad idea.  Failure, even at its best, is simply an abstraction over exception-as-raised.  For Deferreds, this makes sense; asynchronous errors are similar to, and may be directly caused by, code raising an unexpected exception.  However, although Tubes are analogous, a raised exception is an anomalous way for a flow to terminate, and I think the right thing to do there is actually to unconditionally log the exception, passing only a token value saying "application error" into the flowStopped method itself; the main use-case for the "reason" parameter is to distinguish between subtly different protocol conditions which may cause a connection termination; it's not an appropriate place to start handling unexpected exceptions.  (When I have written code that does this using Twisted the inevitable result is an exception whose traceback disappears mysteriously and makes debugging a pain.)

To sum up, the more this abstraction decouples from other things within Twisted, the more it becomes depended upon rather than itself doing the depending, the cleaner, simpler, and easier to use it gets.

So now I think I was wrong.  I now agree with Jean-Paul, and in fact the pendulum of my opinion has swung a bit farther in the other direction.  Not only should Tubes be released independently, I think they are a lower-level library that Twisted should eventually depend on.  This library could just as easily have adapters for other loops, like tornado and asyncio, as well as providing potentially interesting support for doing things like eventually replacing Deferreds - see for example <https://pypi.python.org/pypi/effect>.

So rather than have a single "tubes" package, I think I will break the branch into two pieces:

The "tubes" package itself, which will have no external dependencies and will pretty much just be buffer-management, fan-in, fan-out, buffering, throttling, filtering, routing, and framing-protocol code, intended to remain separate for the long term.  This will probably mean some small amount of code duplication between LineReceiver et. al. and tubes.framing, but that seems fine to me.

The "twisted+tubes" package, perhaps named "helical" (since that is (A) an adjective that describes something which is (B) twisted and (C) also a tube), that depends on tubes and twisted, includes the integration points for endpoint, deferreds, and protocol, and which may be suitable for merging into Twisted later.

I'm traveling at the moment and might not have much time to work on this in the coming week, so if anyone wants to comment on these plans you have plenty of time :-).

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20141212/dde765e7/attachment.html>