[Twisted-Python] Advise for heavy concurrency

Manlio Perillo manlio_perillo at libero.it
Sat Dec 1 04:54:37 MST 2007


Alec Matusis ha scritto:
>> You should break the loop into small parts.
> 
> Can you explain this?
> Do you mean, use reactor.callLater to process say 100 messages at a time? Or maybe use deferToThread for the whole loop?
> 

There is also twisted.internet.task.cooperate, but I have never used it.

>> You see the memory growing because the Python interpreter will not
>> release the memory to the operating system (however the memory should
>> not grow indefinitely)
> 
> The memory is actually growing indefinitely.
> We run 4 identical servers on one machine. The load is not balanced very symmetrically. 
> So one server can have 4500 connections, while another has 6000. 
> There is a certain threshold on the number of connections, when a server starts to leak memory uncontrollably, 
> and it's not set by the highest usage. For example, a server with 4500 connections remains at 100mb RSS for days, 
> while a server that reaches 6000 connections leaks memory to 2gb in 10 hrs 
> (I did not try beyond that, since the machine starts swapping!)


How big is a message sent to each client?
That is, how much memory requires to hold all these messages in memory?

 > [...]



P.S.:
your mail user agent does not wraps long lines; can you please fix it?





Manlio Perillo




More information about the Twisted-Python mailing list