[Twisted-Python] Degrading under load

[Alec Matusis]

We finally upgraded 1.3 to 2.2 without changing the code. The load shown by
'top' for that twistd process may have dropped form 98-99% to 96-98%,
although this is not clear. In any case, the load reduction was marginal, if
any. 
 
Even though the main source of load is OS poll(), we did expect some
improvement from the different scheduler algorithm in 2.2: we have
setTimeout() on each new connection and resetTimeout() every 90 sec on each
connection when keep-alives from the clients arrive.


-----Original Message-----
From: twisted-python-bounces at twistedmatrix.com
[mailto:twisted-python-bounces at twistedmatrix.com] On Behalf Of Itamar
Shtull-Trauring
Sent: Thursday, March 09, 2006 2:32 PM
To: Twisted general discussion
Subject: RE: [Twisted-Python] Degrading under load

On Thu, 2006-03-09 at 13:49 -0800, Alec Matusis wrote:

> I should add here that in our case, the load was driven not by
> connection/disconnection events, but by the number of established
> connections. When that number was in the vicinity of 5000, system poll()
> became very slow (we run poll reactor).

An epoll-based reactor would probably help significantly in this case.
Also note that Twisted 2.x had some algorithmic speed improvements over
1.3 and should scale better (though that doesn't help with poll() being
a bottleneck).


[Twisted-Python] twisted performance
Itamar Shtull-Trauring itamar at itamarst.org 
Wed Dec 7 10:15:43 MST 2005 

Previous message: [Twisted-Python] twisted performance 
Next message: [Twisted-Python] twisted performance 
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] 

----------------------------------------------------------------------------
----

On Wed, 2005-12-07 at 02:04 -0800, Alec Matusis wrote:
> I am running Twisted 1.3 server with a fairly large number of clients.
> The hardware is two 64 bit 3.0 GHz Xeons with HT, 4GB RAM, and it's on
> 2.6.11 kernel.
> I am using poll reactor.
> Currently, when the number of clients approaches 5000, the "top" shows
> 99% CPU load for twistd process, the event loop slows down and weird
> racing conditions show up. 

Could you try this experiment with Twisted 2.1, ideally with the latest
version of Python? There were a number of algorithmic improvements since
1.3 (the one that comes to mind in this case is the scheduler.)

One way to discover if the OS-level poll() is the problem is to use
oprofile; you should be able to use it to see how much time is spent in
there. Before that however, you'd want to use the Python profiler to
figure out if there's any obvious hotspots.