[Twisted-Python] Advice on doing thousands of simultaneous UDP queries with Twisted...

[Tommi Virtanen]

Mike C. Fletcher wrote:
> Writing a simple asynchronous loop myself (poll on a simple socket, send 
> message from queue when writable, read one into other queue when 
> readable) allowed for doing a few thousand queries simultaneously), with 

A word of warning here. Nothing prevents your outgoing socket from
claiming writeability all the time, and the OS from dropping your
packets due to lack of buffer space. UDP just isn't reliable. Been
there, got the T-shirt.

You might be able to do some hacks with SIOCOUTQ, but even that
depends on the OS. IIRC on linux 2.4 it's not useful for UDP.

Your best course of action is to avoid bursts. Here's a nice and simple
algorithm: use a token bucket filter just like suggested elsewhere in
this thred, but add tokens both per time and per received responses.

Don't actually use the TBF as a filter, or even queue packets before it;
use it to see when you should be generating more requests and when not.
As long as there are tokens in bucket, generate reply and send it. When
bucket is empty, return and try again later. With reactor.callLater(),
add tokens to bucket and do the above processing. When receiving a
packet, add a token to bucket and do the above processing.

The more you get replies, the more work gets done; dropped packets slow
things down, as they should. Time-based token adding is there mostly
to be robust against packet loss; in normal case replies should
be the ones filling the bucket. The trick is to continue work as soon as
possible, based on replies (instead of waiting for all n request to
complete before continuing, or until a timer expires).