[Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests

Igor Katson descentspb at gmail.com
Sat Oct 24 10:21:47 EDT 2009

exarkun at twistedmatrix.com wrote:
> On 01:59 pm, terry at jon.es wrote:
>>> Thanks for quick and great help, Terry and Jean-Paul!
>> You're welcome. If in any doubt about conflicting advice, do what JP 
>> suggests.
>> The reason I pointed you to JP's Cooperator solution was that when you
>> extract the set of links from a page that you also want to fetch, you 
>> could
>> make another generator and submit that to the same cooperator's 
>> coiterate
>> function. That would just schedule the additional getPage calls to be 
>> run.
>> The only wrinkle (I think) in that is that in doing the very last 
>> getPage,
>> the cooperator will run out of more work and fire the deferred you got
>> originally. But the final page may contain more links. I'm not sure if 
>> I'm
>> thinking clearly about that final bit. That's your job :-)
> cooperator and DeferredSemaphore are probably both good solutions. 
> There's not much reason to prefer one over the other. :)
> Jean-Paul

That list seems like far from the most active one, but the one filled 
with friendly people :) The thing I like when dealing with Twisted and 
it's documentation is that it is seen often that people who make it, and 
who use it, really love Twisted.

More information about the Twisted-web mailing list