[Twisted-Python] epoll reactor problems

[Alec Matusis]

>  Would it be possible to replicate this issue with your application in some kind of load-testing environment so that we can analyze it "in the lab"?  

 

So far, we could not replicate this in the lab- with thousands of fake multithreaded clients the behavior is normal.

Thus far, we have run this for 2 days with twisted 2.5 with poll and epoll, and we are now seeing big memory leaks that did not occur on 2.2, especially on epoll, in addition to 99.9% CPU.

Back on 2.2 one of our servers was slowly leaking memory, going from 45MB to about 500MB in about 10 days. On epoll in 2.5 it went from 45MB to 1.9g in a couple of hours, almost taking the machine down. The other server did not have memory leak on 2.2, staying at 70MB for weeks at a time. In one day it went from 70MB to 80MB on epoll in 2.5. Both show 99% CPU, so I suspect the memory leaks may be caused by skipping garbage collection because the process is too busy. Interestingly, the actual performance on epoll is not bad, the servers remain responsive even where the poll-based reactor in 2.2 was having difficulties.

 

We must revert to 2.2 tonight.

 

I emailed you and Jean-Paul. If you like, we can arrange ssh to our live servers, then you could look at it on Twisted 2.2 first, then we will switch again to 2.5 and then you can examine it when it’s unstable.

 

 

 

From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-bounces at twistedmatrix.com] On Behalf Of glyph at divmod.com
Sent: Wednesday, April 11, 2007 12:28 PM
To: Twisted general discussion
Subject: RE: [Twisted-Python] epoll reactor problems

 

On 06:38 pm, matusis at matusis.com wrote:

>I wonder if we should revert to Twisted 2.2 ASAP

I can't speak to your deployment requirements, but it sounds like this inquiry could lead to some extremely valuable information to fix this issue in the next release of Twisted.  Would it be possible to replicate this issue with your application in some kind of load-testing environment so that we can analyze it "in the lab"?  Although I'd really like to see this through to a satisfying conclusion, I don't want your application to have to pay the cost of fixing Twisted.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20070412/7e9871d4/attachment.htm