[Twisted-web] Is there a way to implement a deferred sequence?

Andrew Bennetts andrew at bemusement.org
Fri Apr 10 09:28:00 EDT 2009

Alexey S. wrote:
> On Fri, Apr 10, 2009 at 08:43:39AM -0000, glyph at divmod.com wrote:
> >
> > Because Deferreds are intended to be an asynchronous mechanism for  
> > control flow.  They are the way to return a value which is not yet  
> > present.  In other words they are an asynchronous version of exit  
> > semantics; "return" and "raise".
> >
> > They are very explicitly *not* designed to simultaneously implement  
> > asynchronous versions of every other potential control flow - looping,  
> > for example, which seems to be what you want.
> Yes, but there is no synchronous version of what I want, I can't name any.
> Asynchronousness adds some more capabilities to be introduced into flow control.
> >From my perception Deferred is a point where two entities exchange events
> (one send, other receive)

A Deferred's purpose is to provide a mechanism for one entity to communicate a
single result to another.  You're asking for it to do something else, therefore
(by definition, basically) you're asking for something other than a Deferred.

What Deferred provides is pretty similar to just passing in a callback to an
async function, except it's much more convenient.  And like passing callbacks
directly it's a building block for richer control flow, not an all-in-one answer
to every problem.

> And if we think about Deferred as an event router, then my question was
> "why there is only two types of events possible and both of them are 'final' events?"

If you need more events, you can make multiple Deferreds.  in your case you
wanted an event to fire when a set of work has completed — ok, that's one
Deferred right there, with a simple object to track the outstanding work and
fire that Deferred when the amount of remaining work is zero.  Each of those
individual items of work may also have a corresponding Deferred to report their
individual results.

Or as Glyph says, you may simply want to have an object that is called directly
every time some event happens.  Or some combination of the above.

> I just asked about subclassing that event router to include capability
> to route some other events in addition to two existing.
> It is worth noting, that such improvement does not forbid Deferred from being
> "the way to return a value which is not yet present". It just make the term
> "value" more complex, allows it to be dispersed in time, not only concentrated
> at a single moment. I think there is no natural analogy for synchronous flow
> control and can't be - results are not synchronous.
> Perhaps the thing I am talking about should not be called "Deferred", since
> it is only asynchronous version of syncronous flow control. But for me
> extending Deferred looks natural, that is why I am asking.

No, Deferred is very definitely not about multiple events (which is what
multiple values over time are, unless you only want to deal with a final,
single, aggregated value).  Deferred may be a useful building block for a the
thing you are talking about, but it is definitely not a natural extension of it.

> > There are other mechanisms for doing those things.  For example, if you  
> > want to get a stream of data and handle it incrementally, you want to  
> > implement an IProtocol or ITransport.  Trying to cram all of IProtocol  
> > into Deferred is a mistake.
> IProtocol and ITransport looks like being a byte-oriented interfaces, not
> an arbitrary-object-oriented.

They were just an example.  It's not hard to imagine examples using richer
objects than bytes, e.g. with methods like “recordReceived”, “accountCreated” or
“buttonPressed” or “cpuOnFire” rather than “bytesReceived”.  The interesting
part of e.g. IProtocol is that “bytesReceived” can be (and usually is) called
many times rather than once.

By the way, I can imagine a similar transformation to passing in an IProtocol
class, analagous to the transformation of replacing passing in a callback
function with making that function return a Deferred which you then add the
callback to.  So instead of e.g.:

    clientCreator = ClientCreator(reactor, MyProtocolClass)
    d = clientCreator.connectTCP(host, port)

I suppose you could avoid passing the protocol up front:

    clientCreator = HypotheticalDeferringClientCreator(reactor)
    d = clientCreator.connectTCP(host, port)
    def connectionMade(transport):
        protocolInstance = MyProtocolClass()
        return protocolInstance

But really that's just reinventing twisted.internet.protocol.Factory in a less
convenient form...

My point here is basically just that there isn't a single API or design pattern
that is best in every situation.

> I am asking for an half-duplex Event-pipe which can be closed from the sending end.
> Just as variant it can be used to deliver cumulative results, but not fixed to that.
> I said "Deferred" because the implementation of such thing will be identical to
> Deferred's. It will be a Deferred with additional callback(s) which is(are) not
> restricted to be invoked only once. 

So, “identical” except for some fundamental bits that are radically different? :)

Seriously, the extra complexity of the semantics you want is way beyond the
scope of what's reasonable for Deferred.  Handling multiple events is vastly
more complicated in the general case than handling a single result (consider for
instance the various options on DeferredList, and still that API can't even
handle common situations like a dynamically sized list of results, as you've
noticed).  That's not to say that what you want is unreasonable, just that what
you want isn't called “Deferred”.  Deferred does one thing, and does it very


More information about the Twisted-web mailing list