[Twisted-Python] Deferred Groups?

Mike C. Fletcher mcfletch at rogers.com
Sat Jan 21 08:55:01 EST 2006

Duncan McGreggor wrote:

> I have a question about an approach I used... I'm worried that I've 
> over-worked it and have over-looked a more elegant and standard solution.
> I have the need to fire off network connections in groups. Deferreds 
> added to a DeferredList don't fit the bill (because there's no control 
> over all the deferreds in the list). As an example, if you wanted to 
> make a whole batch of concurrent connections, but didn't want to incur 
> the overhead of firing off more than 20 simultaneous connections, 
> you'd split your destination hosts up into groups of 20. As a group 
> was completed, a callback could fire off the next group, etc.

This functionality (or, something very close) is in the deferred module 
already, it's just non-obvious how you use it:

sem = defer.DeferredSemaphore(count)
dl = [sem.run(callable, item, *args, **named) for item in iterable]
return defer.DeferredList(dl, consumeErrors=True)

which, rather than "batching" (which means that you have 3, then 2, then 
1 in play, then 3, then 2, then 1), keeps "count" items in play at any 
time (as long as there are available items), replacing each item that 
completes with one pending item.  I wrap that bit of code with a little 
function called parallel in my code so I don't have to remember the 
trick every time.


  Mike C. Fletcher
  Designer, VR Plumber, Coder

More information about the Twisted-Python mailing list