[Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests

[exarkun at twistedmatrix.com]

On 01:59 pm, terry at jon.es wrote:
>>Thanks for quick and great help, Terry and Jean-Paul!
>
>You're welcome. If in any doubt about conflicting advice, do what JP 
>suggests.
>
>The reason I pointed you to JP's Cooperator solution was that when you
>extract the set of links from a page that you also want to fetch, you 
>could
>make another generator and submit that to the same cooperator's 
>coiterate
>function. That would just schedule the additional getPage calls to be 
>run.
>
>The only wrinkle (I think) in that is that in doing the very last 
>getPage,
>the cooperator will run out of more work and fire the deferred you got
>originally. But the final page may contain more links. I'm not sure if 
>I'm
>thinking clearly about that final bit. That's your job :-)

cooperator and DeferredSemaphore are probably both good solutions. 
There's not much reason to prefer one over the other. :)

Jean-Paul