[Twisted-Python] Degrading under load
matusis at matusis.com
Sat Mar 25 01:36:13 EST 2006
We finally upgraded 1.3 to 2.2 without changing the code. The load shown by
'top' for that twistd process may have dropped form 98-99% to 96-98%,
although this is not clear. In any case, the load reduction was marginal, if
Even though the main source of load is OS poll(), we did expect some
improvement from the different scheduler algorithm in 2.2: we have
setTimeout() on each new connection and resetTimeout() every 90 sec on each
connection when keep-alives from the clients arrive.
From: twisted-python-bounces at twistedmatrix.com
[mailto:twisted-python-bounces at twistedmatrix.com] On Behalf Of Itamar
Sent: Thursday, March 09, 2006 2:32 PM
To: Twisted general discussion
Subject: RE: [Twisted-Python] Degrading under load
On Thu, 2006-03-09 at 13:49 -0800, Alec Matusis wrote:
> I should add here that in our case, the load was driven not by
> connection/disconnection events, but by the number of established
> connections. When that number was in the vicinity of 5000, system poll()
> became very slow (we run poll reactor).
An epoll-based reactor would probably help significantly in this case.
Also note that Twisted 2.x had some algorithmic speed improvements over
1.3 and should scale better (though that doesn't help with poll() being
[Twisted-Python] twisted performance
Itamar Shtull-Trauring itamar at itamarst.org
Wed Dec 7 10:15:43 MST 2005
Previous message: [Twisted-Python] twisted performance
Next message: [Twisted-Python] twisted performance
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Wed, 2005-12-07 at 02:04 -0800, Alec Matusis wrote:
> I am running Twisted 1.3 server with a fairly large number of clients.
> The hardware is two 64 bit 3.0 GHz Xeons with HT, 4GB RAM, and it's on
> 2.6.11 kernel.
> I am using poll reactor.
> Currently, when the number of clients approaches 5000, the "top" shows
> 99% CPU load for twistd process, the event loop slows down and weird
> racing conditions show up.
Could you try this experiment with Twisted 2.1, ideally with the latest
version of Python? There were a number of algorithmic improvements since
1.3 (the one that comes to mind in this case is the scheduler.)
One way to discover if the OS-level poll() is the problem is to use
oprofile; you should be able to use it to see how much time is spent in
there. Before that however, you'd want to use the Python profiler to
figure out if there's any obvious hotspots.
More information about the Twisted-Python