<br><br><div><span class="gmail_quote">On 1/21/06, <b class="gmail_sendername">Duncan McGreggor</b> <<a href="mailto:duncan.mcgreggor@gmail.com">duncan.mcgreggor@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I have a question about an approach I used... I'm worried that I've<br>over-worked it and have over-looked a more elegant and standard<br>solution.<br><br>I have the need to fire off network connections in groups. Deferreds
<br>added to a DeferredList don't fit the bill (because there's no control<br>over all the deferreds in the list). As an example, if you wanted to<br>make a whole batch of concurrent connections, but didn't want to incur<br>
the overhead of firing off more than 20 simultaneous connections, you'd<br>split your destination hosts up into groups of 20. As a group was<br>completed, a callback could fire off the next group, etc.<br><br>What's more, I didn't want to put this kind of control in a factory or
<br>a protocol. In my mind, that didn't seem the proper place for it...</blockquote><div><br>This is one approach. It has the characteristic that if one site in your group is considerably slower than the others, you will wait till all the sites in your group are finished before firing off the next group. This may or may not be good thing for your particular app.
<br><br>An alternative is to create a "pool" of connections that will consume from a queue of potential connections. you feed your list into a DeferredQueue, and create as many concurrent connection handlers as you want, that will all consume from that same queue. this has the characteristic that as long as you keep the queue full you are constantly running 20 connections. This may or may nto be an advantage in the case of your application.
<br><br>Or if you want to use the built-in twisted magic, take a look at twisted.protocols.policies.ThrottlingFactory and other similar things int he same package see if one can be adapted to your use.<br><br>Keep in mind that twisted is not *actually* concurrent, so you may not need to throttle your connections that much, you might be able to let the reactor handle the connection load itself.
<br><br>Actually, given that the reactor handles a thread pool size, is there an equivalent "connection pool size" that can be manipulated from inside the programme? Does such a concept have any use or meaning?<br>
<br>Moof - not a reactor expert, as you can see.<br><br>Moof<br></div></div>