[Twisted-Python] Distributing calls over a time period

Mike Meyer mwm-keyword-twisted.c54178 at mired.org
Tue Jul 18 12:53:31 EDT 2006

In <20060718151310.29014.659879015.divmod.quotient.42787 at ohm>, Jean-Paul Calderone <exarkun at divmod.com> typed:
> >> Can you be more specific about what you mean by "taking up to 33%
> >> more time"?
> >
> >When I set things up to run in 60 seconds, it takes anywhere from 61
> >to 80 seconds.
> >
> >> What kind of time units are we talking about here?
> >
> >10s of seconds, with request counts such that we're getting 10s of
> >requests a second.
> This is orders of magnitude out from where you should be having any
> problems.  Can you share a brief example of how you're doing the
> scheduling?

Sure. Here's the start method that's responsible for starting self.counter
requests in self.opts.send seconds:

    def start(self):
        """Start a page fetch."""

        launch = time()
        defer = getPage('http://%s:8080/axis2/services/RadarWebService' %
                        self.host, postdata=payload, method='POST')
        defer.addCallbacks(self.good, self.bad, callbackArgs=(launch,),
        if not hasattr(self, 'started'):
            self.started = launch

        self.counter -= 1
        if self.counter <= 0:
            delay = (self.started + self.opts.send - time()) / self.counter
            if delay < 0:
                delay = 0
            reactor.callLater(delay, self.start)

done uses self.started and time() to calcuate the time that's passed
and print it:

    def done(self):
            print 'Started %d requests in %f seconds.' % \
                  (self.reqs, time() - self.started)

Mike Meyer <mwm at mired.org>		http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

More information about the Twisted-Python mailing list