[Twisted-Python] 100% CPU on high opened descriptors

Glyph Lefkowitz glyph at twistedmatrix.com
Tue Apr 26 17:30:41 EDT 2011


On Apr 26, 2011, at 4:34 PM, Ray Cote wrote:

> ----- Original Message -----
>> From: "Luke Marsden" <luke-lists at hybrid-logic.co.uk>
>> To: "Twisted general discussion" <twisted-python at twistedmatrix.com>
>> Sent: Tuesday, April 26, 2011 2:08:40 PM
>> Subject: Re: [Twisted-Python] 100% CPU on high opened descriptors
>> On Tue, 2011-04-26 at 17:42 +0200, Juan Antonio Ibañez Santorum wrote:
>>> Hello!
>>>   I have a Linux box serving my twisted app. All works ok except
>>> after some days running (where I can see my CPU going to 100%). I
>>> saw
>>> that CPU goes 100% usage when the app reaches to 1024 opened
>>> descriptors (sockets).
> 
> I assume that using a reverse proxy (Apache, nginx, etc.) in front of Twisted removes this problem? 
> (or at least displaces it from being Twisted's problem to being the proxies problem.)
> 
> Is it recommended to always run Twisted behind a proxy?

No.  It's recommended to fix bugs in Twisted that affect you :).  And besides, this problem is not specific to HTTP; it affects any Twisted service that runs on a port, so there are lots of uses of Twisted where you can't run Twisted behind a proxy.

Plus, this affects clients as well; if you have a twisted spidering application, your client connections could get into this same state.  It's possible to handle this sort of attack without any changes to Twisted, by carefully monitoring the number of connections you have open.

Of course most people just crank up their ulimits and forget about it.

> We've been doing that simply to offload delivery to the final client, but I'm thinking it may have a more general need.

If you have to do it anyway then the discussion is somewhat academic, isn't it? :)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110426/d23de324/attachment.htm 


More information about the Twisted-Python mailing list