[Twisted-Python] Consensus on speed of deferred call/errback-style execution?

[glyph at divmod.com]

On 03:25 pm, terry at jon.es wrote:
>Last year someone mentioned to me that using defer.inlineCallbacks 
>(versus
>writing a bunch of independent standalone callback functions) incurs 
>some
>sort of (significant) speed penalty during execution.
>
>How accurate is that?

I don't believe there's been any extensive profiling of inlineCallbacks. 
At least, none that I'm aware of.  I think what you are remembering is 
that inlineCallbacks generally results in an (often implicit, sometimes 
unintentional) loss of parallelism.  For example, consider this 
idiomatic code:

    xd = foo()
    yd = foo()
    zd = foo()
    def myCallback(x, y, z):
        doStuff(x) + doStuff(y) / doStuff(z)
    gatherResults([x, y, z]).addCallback(myCallback)

That does all three 'foo' operations in parallel, which is generally 
faster.  But the idiomatic inlineCallbacks version:

    doStuff(yield foo()) + doStuff(yield foo()) / doStuff(yield foo())

while making all kinds of sexy use of expressions rather than 
statements, loses that parallelism: you don't get to the second foo() 
call until the first one has completed and its results have been 
processed.  If each foo() call has a 500ms latency this can really add 
up.  Of course you can work around this:

    xd = foo()
    yd = foo()
    zd = foo()
    doStuff(yield xd) + doStuff(yield yd) / doStuff(yield zd)

but it can be difficult to remember to do so, and it starts looking a 
lot more like regular callback-style code.

I think inlineCallbacks is neat, but its strength is really operations 
that are truly conversational; where the calling end of the conversation 
actually does need to wait for each step of an asynchronous conversation 
to complete before moving on to the next one.  A dead giveaway that it's 
going to be awkward to make your inlineCallbacks appropriately parallel 
is if you start writing a 'for' loop that yields a Deferred in its body.