[Twisted-Python] Re: Adding callback to a Deferred that's already running?

David Bolen db3l at fitlinxx.com
Thu Aug 26 15:09:48 EDT 2004

Steve Freitas <sflist at ihonk.com> writes:

> Hi all,
> I've got an XML-RPC app that has to grab some data from a SQL database, then 
> depending on the results, make some more SQL calls, make some more decisions, 
> then finally return a result. So, I'm hip-deep in Deferreds. So, here's the 
> idiom I've come up with, and I'm running into a limitation with it, namely 
> that I can't seem to add callbacks to a Deferred after it has started 
> running. Here's what I'm doing in abbreviated form, and it's an idiom I'd 
> like to stick with, if possible:

You can certainly add a callback to a Deferred that has already fired
(it will immediately run synchronously), but yes, you can't do it
during the callback/errback operation (e.g., adding to a Deferred from
within one of that Deferred's callbacks/errbacks as it is running).

> class MyXmlRpcClass(xmlrpc.XMLRPC):
>   def__init__(self):
>     self.db = adbapi.ConnectionPool(blah)
>   def xmlrpc_foo(self):
>     return FooClass(self.db).step1() # This returns a deferred, see below
> class FooClass(General):
>     def __init__(self, db):
>         self.db = db
>     def step1(self):
>         self.d = self.db.runQuery(blah)
>         self.d.addCallback(self.step2)
>         return self.d
>     def step2(self, query):
>  if query == 'yadda':
>           return 'This bit works'
>         else:
>           self.d.chainDeferred(self.db.runOperation(blah))
>           self.d.addCallback(self.step3)
>     def step3(self, data):
>         return "Never gets here!"
> If I get to the part where it adds the callback for step3, it ends up giving 
> an AlreadyCalled exception in Deferred. So, I expect there's a good way to 
> add to a running Deferred, no?

I don't think you actually want to "add to a running Deferred" here,
since even if your chainDeferred call worked, it would be chaining at
the _end_ of the Deferred's callback/errback chain, which might be
after a whole slew of other operations that your callers (who got the
deferred from step1) already added.  So it wouldn't occur in the
sequence you want anyway.

What you really want to do (I believe) is the opposite chain operation
- you want to take the Deferred from the runOperation(blah), and chain
it to your current Deferred, so that your current Deferred's
callback/errback chain waits on the runOperation result to continue
working.  The actual chaining is easy (just change your use of
chainDeferred), but the hard part is making the current callback chain
suspend itself until the new Deferred finishes.

Luckily, Twisted already provides explicit support for this behavior
(which is sort of crucial to permit deferrable operations to interact
properly anyway).  What I think you want (similar to what I showed in
an earlier message of mine in this thread) is simply to return the new
deferred from within your callback.  Twisted will automatically notice
that the callback itself is awaiting a response via the deferred, and
it establishes the necessary linkage. 

Once the new internal Deferred finishes, it is its result (whether
successful or a Failure object) that will gate whether the original
Deferreds chain continues up callback or errback.

You can peek at the _runCallbacks inside the Deferred class definition
if you want to see the mechanism.  Or check out the first paragraph in
the "Chaining Deferreds" section of the Using Deferreds HOWTO.
Interestingly enough, the HOWTO (next paragraph) makes mention of it
potentially being confusing but that the reader will probably
recognize the need when they run into it.  Clearly that doesn't always
happen :-)

There's two ways, I think you can do FooClass, depending on how you
want to conceptually insert step3 into the processing:

The first, fairly literal adjustment of your class would be:

[Case 1]

class FooClass(General):
    def __init__(self, db):
        self.db = db

    def step1(self):
        return self.db.runQuery(blah).addCallback(self.step2)

    def step2(self, query):
        if query == 'yadda':
            return 'This bit works'
            return self.db.runOperation(blah).addCallback(self.step3)

    def step3(self, data):
        # 'data' is result of self.db.runOperation(blah)

        # Note that the result of this method will become the callback
        # result of the self.db.runOperation(blah) Deferred, and thus
        # in turn the next result passed up the callback chain for the
        # first self.db.runQuery(blah) Deferred as the answer from step2
        # in the non-yadda case.

        return "I got here!'


[Case 2]

class FooClass(General):
    def __init__(self, db):
        self.db = db

    def step1(self):
        d = self.db.runQuery(blah)
        return d.addCallback(self.step2).addCallback(self.step2)
    def step2(self, query):
        if query == 'yadda':
            return 'This bit works'
            return self.db.runOperation(blah)

    def step3(self, data):
        # 'data' is result of step2

        # Note that the result of this method will become the callback
        # in the main callback chain for the first self.db.runQuery(blah)
        # Deferred, and will thus replace that of step2
        return "I got here!'

(Actually, there's also a third way in which you manually pause and
unpause the main Deferred callback chain, and set up a callback on the
secondary Deferred chain to resume the main one, but that's precisely
what Twisted does for you when you just return the underlying Deferred
from your callback, so I don't see any benefit of doing it yourself).

In both of these cases there are two deferred chains running (arising
from the two discrete deferrable operations that themselves are
creating Deferreds), but that in the even of the non-yadda path in
step2, the first callback chain is suspended while awaiting completion
of the second.

In Case 1, the addCallback of step3 is to the secondary callback
chain, so it provides additional post-processing of the runOperation
result, but does not get involved in other paths/results from step2.

In Case 2, step3 is added to the main callback (runQuery) chain, so it
will see any results of step2 (both yadda and runOperation) and can
then post-process them, becoming in all cases the callback result for
the main callback chain.  Note a key point here - even though step3 is
next in the main callback chain after step2, in the event step2
returns the Deferred from runOperation, the main chain suspends (so
step3 doesn't run immediately) until that second Deferred finishes,
at which point it is the result that step3 sees.

So what you end up (in the non-yadda case) is something like (in poor
ASCII diagraming):

    Case 1:                           Case 2:

    runQuery                          runQuery
       |                                 |
    step2                             step2
       |                                 |
       |  no                             |  no
       +-yadda->-runOperation            +-yadda->-runOperation
       |              |                  |              |
    yadda             v               yadda             v
       |              |                  |              |
       +---<------ step3                 +---<----------+
       |                                 |
       v                              step3

Hope this helps clarify things somewhat.

-- David

More information about the Twisted-Python mailing list