[Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests

Valentino Volonghi dialtone at gmail.com
Tue Oct 27 14:43:00 EDT 2009

On Oct 24, 2009, at 6:20 AM, Igor Katson wrote:

> Is my idea about a "wrapper" practical and what's the general way to
> write it?
> On which event is it better to decrement the counter of the amount
> currently downloading files?

Another way that you might do this is by using this small snippet:


turtl is a project that you can use either as a proxy server in front
of all of your clients (and throttle the requests on a url basis) or
embedded in your system like this:

                 from turtl import engine
                 thr = engine.ThrottlingDeferred(parallelism, calls,  
                 dl = [thr.run(callable, *args, **kwargs) for args in  
                 defer.DeferredList(dl).addBoth(lambda _: reactor.stop 

In this case you get a lot more control over the number of calls in a  
interval that you are allowed to make (for example Amazon Alexa allows
only 15 calls per second).

Valentino Volonghi aka Dialtone
Now Running MacOSX 10.6

More information about the Twisted-web mailing list