[Twisted-Python] Advise for heavy concurrency

Manlio Perillo manlio_perillo at libero.it
Sat Dec 1 06:54:37 EST 2007

Alec Matusis ha scritto:
>> You should break the loop into small parts.
> Can you explain this?
> Do you mean, use reactor.callLater to process say 100 messages at a time? Or maybe use deferToThread for the whole loop?

There is also twisted.internet.task.cooperate, but I have never used it.

>> You see the memory growing because the Python interpreter will not
>> release the memory to the operating system (however the memory should
>> not grow indefinitely)
> The memory is actually growing indefinitely.
> We run 4 identical servers on one machine. The load is not balanced very symmetrically. 
> So one server can have 4500 connections, while another has 6000. 
> There is a certain threshold on the number of connections, when a server starts to leak memory uncontrollably, 
> and it's not set by the highest usage. For example, a server with 4500 connections remains at 100mb RSS for days, 
> while a server that reaches 6000 connections leaks memory to 2gb in 10 hrs 
> (I did not try beyond that, since the machine starts swapping!)

How big is a message sent to each client?
That is, how much memory requires to hold all these messages in memory?

 > [...]

your mail user agent does not wraps long lines; can you please fix it?

Manlio Perillo

More information about the Twisted-Python mailing list