[Twisted-Python] twisted performance

Alec Matusis matusis at matusis.com
Wed Dec 7 05:04:26 EST 2005


I am running Twisted 1.3 server with a fairly large number of clients.
The hardware is two 64 bit 3.0 GHz Xeons with HT, 4GB RAM, and it's on
2.6.11 kernel.
I am using poll reactor.
Currently, when the number of clients approaches 5000, the "top" shows 99%
CPU load for twistd process, the event loop slows down and weird racing
conditions show up. 
 
My clients basically do nothing: they connect and just sit there, sending
application-level keep-alives (null character string) to the server every 90
sec, which are echoed back. The connections/ disconnections for clients are
rare- about 4-5 per sec. So I am mostly dealing with 5000 clients in the
ESTABLISHED state. Very rarely (5x per second on average) one of the clients
gets a message from the server, but I am sure that this is not what keeps
the load so high (I slowed down message rates, and the CPU load did not
change).  
 
What are your thoughts on this performance? Is this load mainly due to
poll() overhead, polling a large number of file descriptors? 
I would love to try an epoll reactor, but as I understand, it's not yet
available?
 
Additionally, the process leaks about 100MB memory per day. I made the
garbage collector print out the number of objects traced by it every 30 min,
and this number stays more or less constant. When the memory held by the
process becomes larger, the event loop further slows down, and the
performance with the same number of clients degrades, as the memory
footprint increases. 
 
Thanks for any advice.
 
Alec.
 


 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20051207/3b5880f5/attachment.htm 


More information about the Twisted-Python mailing list