[Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests

[Terry Jones]

> Thanks for quick and great help, Terry and Jean-Paul!

You're welcome. If in any doubt about conflicting advice, do what JP suggests.

The reason I pointed you to JP's Cooperator solution was that when you
extract the set of links from a page that you also want to fetch, you could
make another generator and submit that to the same cooperator's coiterate
function. That would just schedule the additional getPage calls to be run.

The only wrinkle (I think) in that is that in doing the very last getPage,
the cooperator will run out of more work and fire the deferred you got
originally. But the final page may contain more links. I'm not sure if I'm
thinking clearly about that final bit. That's your job :-)

Terry