[Twisted-Python] Advice on doing thousands of simultaneous UDP queries with Twisted...
Mike C. Fletcher
mcfletch at rogers.com
Tue Feb 10 02:24:04 EST 2004
Stephen Thorne wrote:
>Have a look at htb.py, you should be able to rate limit the udp
>throughput rather easily.
>For me doing web stuff it was (in its entirety) like this:
<bandwidth throttler clipped>
>Hope that helps.
If for no other reason than that it convinced me there wasn't a simple
and elegant solution readily available to be dropped in for a UDP system
:) . In the final analysis, what was happening is that individual
queries were timing out *before the queries were even sent* because it
was taking so incredibly long to format and send all of the queries.
The manually written system was "pull" based. It marked the
time-of-sending for timeout purposes deep in the select loop. The
Twisted version is setting the timeout via a callLater when the request
is *submitted*. With 2 or 3 thousand queries, the 20 seconds of timeout
was finishing before any messages were processed from the incoming port,
then the timeouts would all get called, which kept eating up processing
time, preventing any messages from getting read, so the port's internal
buffer would fill up and more timeouts occurred, so basically nothing
would get through.
Queueing up the outputs before sending didn't help, btw. Nor would
simple bandwidth throttling without modifying the timeout characteristics.
My solution, which is rather low tech, is to do an effective "yield"
after each message is sent, to allow for processing incoming responses.
This is done with this little method:
def smallBatch( self, oids, tables, index=0, iterDelay=.01 ):
if index < len(self.proxies):
proxy = self.proxies[index]
reactor.callLater( iterDelay, self.smallBatch, oids, tables,
dl = defer.DeferredList( self.partialDefers )
dl.addCallback( self.returnFinal )
(which also has a side-effect of introducing a bandwidth throttling),
but most importantly, it delays setting the timeout on any given query
until when the query is actually sent (or fairly close to then). Put
another way, it gives the system a chance to send the message as soon
after submitting it as possible.
Have fun all, and thanks, Stephen,
>On Mon, 2004-02-09 at 20:51, Mike C. Fletcher wrote:
>>only a few dozen dropped messages. However, with the Twisted equivalent
>>(UDPTransport with my simple protocol object), I was seeing huge drop
>>rates, so, gathering that Twisted isn't queueing up the UDP requests, I
>>wrote a (byzantine) query-throttling mechanism with Twisted defers.
Mike C. Fletcher
Designer, VR Plumber, Coder
More information about the Twisted-Python