[Twisted-Python] How to "monitor" parallel executions of Deferreds

[Jason Fritcher]

Pedro Sanchez wrote:
> Hello,
> 
> I need to implement a function (say, myfunc) that fires several
> independent Deferreds (using deferToThread), and that returns a deferred
> itself. A client of this function (say, client) is expected to do
> something like this:
> 
> def client():
>     def done(success)
>         if success:
>            print 'd1 and d2 are done'
> 
>     d = myfunc()
>     d.addCallback(done)
>     ...
> 
> 
> myfuct() above does something like this:
> 
> def myfunc():
>     def checkFunc1(success):
>     ...
> 
>     def checkFunc2(success):
>     ...
> 
>     d = defer.Deferred()
>     d1 = deferToThread(func1).addCallback(checkFunc1)
>     d2 = deferToThread(func2).addCallback(checkFunc2)
>     d.addCallback(????)
> 
>     return d
> 
> Clearly d, d1, and d2 in the above are disconnected. And I don't want to
> serialize d1 and d2 which would be a way to connect them all.
> 
> The problem is I don't know how d should be defined. The idea is that
> the 'done' function in the client function is only called when both
> threads d1 and d2 are done.

I think what you're looking for is DeferredList. You pass it a list of
deferreds you want to wait on and DeferredList will only fire its
callback method once all of the other deferred are complete. Read the
docs because the first callback you add to it will get a list of results
from all the defereds.

So for your above code, you'd want to do something like...

d1 = deferToThread(...)
d2 = deferToThread(...)
dl = defer.DeferredList([d1, d2])
dl.addCallback(...) # If needed here.
return dl

Just be aware to not add any additional callbacks to d1 or d2 after you
add them to the deferredlist.

Hope this helps. :)

-- 
Jason Fritcher
Software Engineer
Core Infrastructure Services & Strategy
Earthlink, Inc
fritcher at corp.earthlink.net
(404) 748-7262, x22262
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20050825/f1fd0150/attachment.pgp