[Twisted-Python] Advice on doing thousands of simultaneous UDP queries with Twisted...
Mike C. Fletcher
mcfletch at rogers.com
Mon Feb 9 05:51:45 EST 2004
I've written a UDP-based protocol adaptor (TwistedSNMP) where one of the
key requirements is the ability to scan thousands of SNMP Agents
"simultaneously" (i.e. the requests should be asynchronously sent and
retired as the Agents respond).
Writing a simple asynchronous loop myself (poll on a simple socket, send
message from queue when writable, read one into other queue when
readable) allowed for doing a few thousand queries simultaneously), with
only a few dozen dropped messages. However, with the Twisted equivalent
(UDPTransport with my simple protocol object), I was seeing huge drop
rates, so, gathering that Twisted isn't queueing up the UDP requests, I
wrote a (byzantine) query-throttling mechanism with Twisted defers.
Problem is, it's a byzantine, fragile (and *slow*) solution to what
would *seem* to be one of the most common requirements in networked
development. Worse yet, because I am seeing such high drop rates I wind
up having to batch in very small groups, serially (instead of
parallel-ly), so the primary purpose of the system (fast querying of
thousands of agents) is lost. (Instead of taking 1 or 2 minutes to
query 800 or so Agents it will take on the order of 10 minutes.)
So, the question: is there a simple way to turn on buffered mode in UDP
transports so that they can deal with queueing up a few thousand
messages to send, sending them, then having a few thousand computers
send a reply (within a few seconds of one another)? Is Twisted just not
Queue-reliant via use of some mechanism I haven't discovered yet? Even
if I do find a decent queueing mechanism, I'm still left with the
problem that timeouts and the like are going to wind up being measured
from queueing-time, rather than sending time... not an issue if
everything gets sent in a half-second or so, but a real problem if it
takes 8 or 9 seconds just to send the original messages out.
Looking at the udp.Port class, I'm not seeing anything providing a
queue, seems as though there's a non-blocking write or read, but nothing
to handle overflows of sends or receives AFAICT, though it looks as
though a protocol could do some queueing on incoming in its
datagramReceived... just not sure how that would work.
Mike C. Fletcher
Designer, VR Plumber, Coder
More information about the Twisted-Python