[Twisted-Python] Degrading under load

Yitzchak Gale gale at sefer.org
Thu Mar 9 16:13:21 EST 2006


Sorry, I guess my question wasn't clear enough.

The most important things I need to know are:

When running listenTCP, how often does twisted
accept pending connections on the port? Is it only
when the previous connection is finished
processing, or every time the event loop gets
control, or something in between?

And when twisted does accept pending connections,
does it accept ALL of them and queue them all for
processing, or just one at a time?

Thanks,
Yitz

My original post:
> I need to set up a TCP service (on a linux box)
> that will get something like a few hunderd connections
> per minute at peak load. For each connection, I do
> some XML processing, and possibly send a query
> to another nearby machine and get a respone.
>
> Seems to me that twisted should be able to handle that.
>
> But what happens when I get the occasional burst
> of connections, lets say tens of connections within
> one second? What I need is:
>
> o Every client gets a socket connection promptly, so
>  no danger of TCP timeout.
> o Under medium load, clients will have to wait a
>  bit longer for the response.
> o Under heavy load, some clients will get a "busy"
>  response (defined in the protocol I am implementing)
>  and immediate socket close.
>
> What is the best way to do that in twisted? I envision
> one of the following architectures:
>
> A. Just use twisted in the usual way. Watch twisted's
> event queue for heavy load.
>
> B. Two processes: one to dish out connections and one
> to queue requests and process them.
>
> C. Three processes: one to dish out connections, one
> to queue requests and watch for load, and one to
> process the requests.
>
> Which of these do I need to use to get the desired
> effect under load? Or is there some better way?




More information about the Twisted-Python mailing list