[Twisted-Python] Deferred Groups?

Duncan McGreggor duncan.mcgreggor at gmail.com
Sat Jan 21 18:04:08 EST 2006


On Jan 21, 2006, at 6:55 AM, Mike C. Fletcher wrote:

> Duncan McGreggor wrote:
>
>> I have a question about an approach I used... I'm worried that I've 
>> over-worked it and have over-looked a more elegant and standard 
>> solution.
>>
>> I have the need to fire off network connections in groups. Deferreds 
>> added to a DeferredList don't fit the bill (because there's no 
>> control over all the deferreds in the list). As an example, if you 
>> wanted to make a whole batch of concurrent connections, but didn't 
>> want to incur the overhead of firing off more than 20 simultaneous 
>> connections, you'd split your destination hosts up into groups of 20. 
>> As a group was completed, a callback could fire off the next group, 
>> etc.
>
> This functionality (or, something very close) is in the deferred 
> module already, it's just non-obvious how you use it:
>
> sem = defer.DeferredSemaphore(count)
> dl = [sem.run(callable, item, *args, **named) for item in iterable]
> return defer.DeferredList(dl, consumeErrors=True)
>
> which, rather than "batching" (which means that you have 3, then 2, 
> then 1 in play, then 3, then 2, then 1), keeps "count" items in play 
> at any time (as long as there are available items), replacing each 
> item that completes with one pending item.  I wrap that bit of code 
> with a little function called parallel in my code so I don't have to 
> remember the trick every time.

Ah, this is fantastic! Thanks for the heads up. I've been buried in the 
deferred file for the paste few days and have no idea how I missed 
this... I'm now reading through the entire file to see what other 
goodies I might have missed...

I've replaced almost all of that test code, and it works like a charm. 
Your usage example was a huge time-saver -- thanks!

d





More information about the Twisted-Python mailing list