[Twisted-Python] Degrading under load
gale at sefer.org
Wed Mar 8 05:46:35 EST 2006
I need to set up a TCP service (on a linux box)
that will get something like a few hunderd connections
per minute at peak load. For each connection, I do
some XML processing, and possibly send a query
to another nearby machine and get a respone.
Seems to me that twisted should be able to handle that.
But what happens when I get the occasional burst
of connections, lets say tens of connections within
one second? What I need is:
o Every client gets a socket connection promptly, so
no danger of TCP timeout.
o Under medium load, clients will have to wait a
bit longer for the response.
o Under heavy load, some clients will get a "busy"
response (defined in the protocol I am implementing)
and immediate socket close.
What is the best way to do that in twisted? I envision
one of the following architectures:
A. Just use twisted in the usual way. Watch twisted's
event queue for heavy load.
B. Two processes: one to dish out connections and one
to queue requests and process them.
C. Three processes: one to dish out connections, one
to queue requests and watch for load, and one to
process the requests.
Which of these do I need to use to get the desired
effect under load? Or is there some better way?
More information about the Twisted-Python