[Twisted-Python] Deferred Groups?

Duncan McGreggor duncan.mcgreggor at gmail.com
Sat Jan 21 18:06:51 EST 2006


On Jan 21, 2006, at 7:01 AM, Moof wrote:

> On 1/21/06, Duncan McGreggor <duncan.mcgreggor at gmail.com> wrote:
>
>> over-worked it and have over-looked a more elegant and standard
>> solution.
>>
>> I have the need to fire off network connections in groups. Deferreds
>> added to a DeferredList don't fit the bill (because there's no control
>> over all the deferreds in the list). As an example, if you wanted to
>> make a whole batch of concurrent connections, but didn't want to incur
>>  the overhead of firing off more than 20 simultaneous connections, 
>> you'd
>> split your destination hosts up into groups of 20. As a group was
>> completed, a callback could fire off the next group, etc.
>>
>> What's more, I didn't want to put this kind of control in a factory or
>> a protocol. In my mind, that didn't seem the proper place for it...
>
> An alternative is to create a "pool" of connections that will consume 
> from a queue of potential connections. you feed your list into a 
> DeferredQueue, and create as many concurrent connection handlers as 
> you want, that will all consume from that same queue. this has the 
> characteristic that as long as you keep the queue full you are 
> constantly running 20 connections. This may or may nto be an advantage 
> in the case of your application.
>
> Or if you want to use the built-in twisted magic, take a look at 
> twisted.protocols.policies.ThrottlingFactory and other similar things 
> int he same package see if one can be adapted to your use.
>
> Keep in mind that twisted is not *actually* concurrent, so you may not 
> need to throttle your connections that much, you might be able to let 
> the reactor handle the connection load itself.
>
> Actually, given that the reactor handles a thread pool size, is there 
> an equivalent "connection pool size" that can be manipulated from 
> inside the programme? Does such a concept have any use or meaning?
>
> Moof - not a reactor expert, as you can see.

But very cool ideas and some great things to explore -- thanks!

d





More information about the Twisted-Python mailing list