[Twisted-Python] defer.gatherResults & Deferreds chaining

Jorge Gonzalez gjorge at google.com
Mon Jul 29 14:29:18 MDT 2013


I'm trying to understand why a chain of Deferreds I'm dealing with hangs
and why return values from some callbacks are not always making it to other
callbacks down the chain of Deferreds.

Is it possible that there could be a race condition between Deferred
chaining and Deferred result
gathering<http://twistedmatrix.com/documents/current/core/howto/defer.html>
?
I.e., Chaining Deferreds<http://twistedmatrix.com/documents/current/core/howto/defer.html#auto13>
says
that "*If you need one Deferred to wait on another, all you need to do is
return a Deferred from a method added to addCallbacks. Specifically, if you
return Deferred B from a method added to Deferred A using A.addCallbacks,
Deferred A's processing chain will stop until Deferred B's .callback()
method is called; at that point, the next callback in A will be passed the
result of the last callback in Deferred B's processing chain at the time*."

Now at the same time,
DeferredList<http://twistedmatrix.com/documents/current/core/howto/defer.html#auto8>
 (on top of which gatherResults is built) warns that "*If you want to apply
callbacks to the individual Deferreds that go into the DeferredList, you
should be careful about when those callbacks are added. The act of adding a
Deferred to a DeferredList inserts a callback into that Deferred ... The
important thing to remember is that it is this callback which records the
value that goes into the result list handed to the DeferredList's callback.*
*Therefore, if you add a callback to the Deferred after adding the Deferred
to the DeferredList, the value returned by that callback will not be given
to the DeferredList's callback. To avoid confusion, we recommend not adding
callbacks to a Deferred once it has been used in a DeferredList*"

Given these two explanations, say I have something like this:
---------------------------------------------------------------------------
def slowFuncReturningDeferred_A():
  ...

def slowFuncReturningDeferred_B():
  ...

d = defer.Deferred()
d.addCallback(labmda _: slowFuncReturningDeferred_A())
d.addCallback(labmda _: slowFuncReturningDeferred_B())

defer.gatherResults([d])
---------------------------------------------------------------------------

Wouldn't it be possible that defer.gatherResults inserts its callback
into d before
say slowFuncReturningDeferred_B gets to run and return its own Deferred
(and therefore chain it to what gatherRestults is ultimately waiting for?).
If so, wouldn't the results returned by slowFuncReturningDeferred_B never
make it to the results gathered by defer.gatherResults?

Thanks,
Jorge
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20130729/63d8b800/attachment.html>


More information about the Twisted-Python mailing list