[Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests
terry at jon.es
Sat Oct 24 09:59:16 EDT 2009
> Thanks for quick and great help, Terry and Jean-Paul!
You're welcome. If in any doubt about conflicting advice, do what JP suggests.
The reason I pointed you to JP's Cooperator solution was that when you
extract the set of links from a page that you also want to fetch, you could
make another generator and submit that to the same cooperator's coiterate
function. That would just schedule the additional getPage calls to be run.
The only wrinkle (I think) in that is that in doing the very last getPage,
the cooperator will run out of more work and fire the deferred you got
originally. But the final page may contain more links. I'm not sure if I'm
thinking clearly about that final bit. That's your job :-)
More information about the Twisted-web