[Twisted-Python] Advise for heavy concurrency
manlio_perillo at libero.it
Sat Dec 1 06:54:37 EST 2007
Alec Matusis ha scritto:
>> You should break the loop into small parts.
> Can you explain this?
> Do you mean, use reactor.callLater to process say 100 messages at a time? Or maybe use deferToThread for the whole loop?
There is also twisted.internet.task.cooperate, but I have never used it.
>> You see the memory growing because the Python interpreter will not
>> release the memory to the operating system (however the memory should
>> not grow indefinitely)
> The memory is actually growing indefinitely.
> We run 4 identical servers on one machine. The load is not balanced very symmetrically.
> So one server can have 4500 connections, while another has 6000.
> There is a certain threshold on the number of connections, when a server starts to leak memory uncontrollably,
> and it's not set by the highest usage. For example, a server with 4500 connections remains at 100mb RSS for days,
> while a server that reaches 6000 connections leaks memory to 2gb in 10 hrs
> (I did not try beyond that, since the machine starts swapping!)
How big is a message sent to each client?
That is, how much memory requires to hold all these messages in memory?
your mail user agent does not wraps long lines; can you please fix it?
More information about the Twisted-Python