[Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Sat Oct 24 10:14:27 EDT 2009

On 01:59 pm, terry at jon.es wrote:
>>Thanks for quick and great help, Terry and Jean-Paul!
>You're welcome. If in any doubt about conflicting advice, do what JP 
>The reason I pointed you to JP's Cooperator solution was that when you
>extract the set of links from a page that you also want to fetch, you 
>make another generator and submit that to the same cooperator's 
>function. That would just schedule the additional getPage calls to be 
>The only wrinkle (I think) in that is that in doing the very last 
>the cooperator will run out of more work and fire the deferred you got
>originally. But the final page may contain more links. I'm not sure if 
>thinking clearly about that final bit. That's your job :-)

cooperator and DeferredSemaphore are probably both good solutions. 
There's not much reason to prefer one over the other. :)


More information about the Twisted-web mailing list