[Twisted-Python] Advice on doing thousands of simultaneous UDP queries with Twisted...
stephen at thorne.id.au
Mon Feb 9 06:32:18 EST 2004
Have a look at htb.py, you should be able to rate limit the udp
throughput rather easily.
For me doing web stuff it was (in its entirety) like this:
bucket = htb.Bucket(parent)
bucket.rate = 4000
bucket.maxburst = 12000
bucketFilter = htb.FilterByHost()
bucketFilter.bucketFactory = RateBucket
application = service.Application("simple")
site_factory = server.Site( Directory() )
site_factory.protocol = htb.ShapedProtocolFactory(
Hope that helps.
On Mon, 2004-02-09 at 20:51, Mike C. Fletcher wrote:
> I've written a UDP-based protocol adaptor (TwistedSNMP) where one of the
> key requirements is the ability to scan thousands of SNMP Agents
> "simultaneously" (i.e. the requests should be asynchronously sent and
> retired as the Agents respond).
> Writing a simple asynchronous loop myself (poll on a simple socket, send
> message from queue when writable, read one into other queue when
> readable) allowed for doing a few thousand queries simultaneously), with
> only a few dozen dropped messages. However, with the Twisted equivalent
> (UDPTransport with my simple protocol object), I was seeing huge drop
> rates, so, gathering that Twisted isn't queueing up the UDP requests, I
> wrote a (byzantine) query-throttling mechanism with Twisted defers.
> Problem is, it's a byzantine, fragile (and *slow*) solution to what
> would *seem* to be one of the most common requirements in networked
> development. Worse yet, because I am seeing such high drop rates I wind
> up having to batch in very small groups, serially (instead of
> parallel-ly), so the primary purpose of the system (fast querying of
> thousands of agents) is lost. (Instead of taking 1 or 2 minutes to
> query 800 or so Agents it will take on the order of 10 minutes.)
> So, the question: is there a simple way to turn on buffered mode in UDP
> transports so that they can deal with queueing up a few thousand
> messages to send, sending them, then having a few thousand computers
> send a reply (within a few seconds of one another)? Is Twisted just not
> Queue-reliant via use of some mechanism I haven't discovered yet? Even
> if I do find a decent queueing mechanism, I'm still left with the
> problem that timeouts and the like are going to wind up being measured
> from queueing-time, rather than sending time... not an issue if
> everything gets sent in a half-second or so, but a real problem if it
> takes 8 or 9 seconds just to send the original messages out.
> Looking at the udp.Port class, I'm not seeing anything providing a
> queue, seems as though there's a non-blocking write or read, but nothing
> to handle overflows of sends or receives AFAICT, though it looks as
> though a protocol could do some queueing on incoming in its
> datagramReceived... just not sure how that would work.
> Thoughts appreciated,
> Mike C. Fletcher
> Designer, VR Plumber, Coder
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
More information about the Twisted-Python