[Twisted-Python] Consensus on speed of deferred call/errback-style execution?
glyph at divmod.com
glyph at divmod.com
Fri Oct 24 12:03:39 EDT 2008
On 03:25 pm, terry at jon.es wrote:
>Last year someone mentioned to me that using defer.inlineCallbacks
>(versus
>writing a bunch of independent standalone callback functions) incurs
>some
>sort of (significant) speed penalty during execution.
>
>How accurate is that?
I don't believe there's been any extensive profiling of inlineCallbacks.
At least, none that I'm aware of. I think what you are remembering is
that inlineCallbacks generally results in an (often implicit, sometimes
unintentional) loss of parallelism. For example, consider this
idiomatic code:
xd = foo()
yd = foo()
zd = foo()
def myCallback(x, y, z):
doStuff(x) + doStuff(y) / doStuff(z)
gatherResults([x, y, z]).addCallback(myCallback)
That does all three 'foo' operations in parallel, which is generally
faster. But the idiomatic inlineCallbacks version:
doStuff(yield foo()) + doStuff(yield foo()) / doStuff(yield foo())
while making all kinds of sexy use of expressions rather than
statements, loses that parallelism: you don't get to the second foo()
call until the first one has completed and its results have been
processed. If each foo() call has a 500ms latency this can really add
up. Of course you can work around this:
xd = foo()
yd = foo()
zd = foo()
doStuff(yield xd) + doStuff(yield yd) / doStuff(yield zd)
but it can be difficult to remember to do so, and it starts looking a
lot more like regular callback-style code.
I think inlineCallbacks is neat, but its strength is really operations
that are truly conversational; where the calling end of the conversation
actually does need to wait for each step of an asynchronous conversation
to complete before moving on to the next one. A dead giveaway that it's
going to be awkward to make your inlineCallbacks appropriately parallel
is if you start writing a 'for' loop that yields a Deferred in its body.
More information about the Twisted-Python
mailing list