[Twisted-web] Limit the simultaneous twisted.web.client.downloadPage requests
Valentino Volonghi
dialtone at gmail.com
Tue Oct 27 14:43:00 EDT 2009
On Oct 24, 2009, at 6:20 AM, Igor Katson wrote:
> Is my idea about a "wrapper" practical and what's the general way to
> write it?
> On which event is it better to decrement the counter of the amount
> currently downloading files?
Another way that you might do this is by using this small snippet:
http://bitbucket.org/adroll/turtl/src/tip/turtl/engine.py
turtl is a project that you can use either as a proxy server in front
of all of your clients (and throttle the requests on a url basis) or
embedded in your system like this:
from turtl import engine
thr = engine.ThrottlingDeferred(parallelism, calls,
interval)
dl = [thr.run(callable, *args, **kwargs) for args in
self.args]
defer.DeferredList(dl).addBoth(lambda _: reactor.stop
())
In this case you get a lot more control over the number of calls in a
given
interval that you are allowed to make (for example Amazon Alexa allows
only 15 calls per second).
--
Valentino Volonghi aka Dialtone
Now Running MacOSX 10.6
http://www.adroll.com/
More information about the Twisted-web
mailing list