[Twisted-Python] Adding callback to a Deferred that's already running?

Abe Fettig abe at fettig.net
Thu Aug 26 12:12:00 EDT 2004


I'm all too familar with the situation you describe, and I've come up 
with a pattern that works well for me.  Here's how I would have written 
your example:

class XMLRPCResponseClass():
     def step1(self):
	finished = defer.Deferred()
             self.step2, finished).addErrback(finished.errback)
         return finished

     def step2(self, result, finished):
         if result == 'yadda':
             finished.callback('This bit works')

It takes a while to really understand how Deferreds work, but I'll try 
to explain this example.  In step1, we create a new Deferred called 
'finished'.  This Deferred didn't come from a method we called; we 
created it from scratch.  This means that it won't be triggered 
automatically when some event finishes.  Instead, we have the freedom to 
call it whenever we want.

We then kick off the asyncronous operation self.db.runQuery.  We set up 
the callbacks as follows:

1. If runQuery completes successfully, call step2(), passing finished as 
an additional argument.
2. If runQuery fails, pass the error on to finished.

Then we return finished, our Deferred.  At this point finished.errback 
will get called if runQuery fails.  But what if runQuery succeeds?  That 
will be handled by step2.

In step2, we take the result of runQuery , as well as the 'finished' 
Deferred we created in step1.  Now we have the results of the first 
operation, as well as a reference to the deferred that was returned in 

If the result of runQuery was 'yadda', we callback finished with a 
successful value.  Otherwise, we kick of a second asyncronous operation, 
(self.d.runOperation) and chain the results to finished.  Remember that 
a.chainDeferred(b) doesn't do anything special, it's just shorthand for:


So the result is that finished ends up calling back or erring back with 
the results of self.d.runOperation, even though it wasn't bound to 
runOperation when it was first created.

The only thing to be careful of with this technique is that you have to 
make sure any operation that could possibly fail sends its error to 
finished.errback - otherwise finished will never be finished :-)

I hope that's helpful!  I'm happy to answer questions if you have any.


p.s. - It's possible there's a better way to do this sort of thing.  But 
I'm using the above technique extensively in Hep and Yarn without any 

Steve Freitas wrote:
>>This wasn't spectacularly obvious to me the first time either, but I 
>>think it's covered somewhere in the deferred execution howto
> Ah ha! I think I figured it out. If I do this, it seems to work...
> class XMLRPCResponseClass():
>     def step1(self):
>         self.d = self.db.runQuery(blah)
>         self.d.addCallback(self.step2)
>         return self.d
>     def step2(self, query): # Here's where the change is
>         if query == 'yadda':
>           return 'This bit works'
>         else:
>           self.d = self.db.runOperation(blah) # Rebinding self.d to new
>                                               # Deferred()
>           self.d.addCallback(self.step3) # add callbacks to new one
>           return self.d # And return ref to the new one
>     def step3(self, data):
>         return 'Now it gets here!'
> The trick is that I need to add any subsequent callbacks to a new
> deferred which I return from the function. Strangely (to me), if I try
> using chainDeferred() instead of returning the new deferred from step2,
> it fails with AlreadyCalled, which I don't yet understand, but that's
> fine, 'cause this seems to work well enough.
> I really like this technique because I don't have to use a big state
> machine at the root level of the xmlrpc calls, or preassign possibly
> unnecessary callbacks, or toss around callables as variables until I'm
> thoroughly confused. Instead, it allows me to place all the
> (client||server) state information for a given method call inside a
> class instance which persists only until the xmlrpc method returns. And
> multiple calls to this method while it's being simultaneously Deferred()
> for other clients don't require any extra coding to handle, as I've got
> one class instance per connection. Very clean, very well separated,
> which is hard to figure out how to do with this wonderful framework.
> Alright, I'll stop ranting now. :-) I still have to give this thing a
> workout to make sure it does all I want it to.
> Thanks for the help!
> Steve
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

More information about the Twisted-Python mailing list