[Twisted-Python] Advise for heavy concurrency
Manlio Perillo
manlio_perillo at libero.it
Sat Dec 1 04:54:37 MST 2007
Alec Matusis ha scritto:
>> You should break the loop into small parts.
>
> Can you explain this?
> Do you mean, use reactor.callLater to process say 100 messages at a time? Or maybe use deferToThread for the whole loop?
>
There is also twisted.internet.task.cooperate, but I have never used it.
>> You see the memory growing because the Python interpreter will not
>> release the memory to the operating system (however the memory should
>> not grow indefinitely)
>
> The memory is actually growing indefinitely.
> We run 4 identical servers on one machine. The load is not balanced very symmetrically.
> So one server can have 4500 connections, while another has 6000.
> There is a certain threshold on the number of connections, when a server starts to leak memory uncontrollably,
> and it's not set by the highest usage. For example, a server with 4500 connections remains at 100mb RSS for days,
> while a server that reaches 6000 connections leaks memory to 2gb in 10 hrs
> (I did not try beyond that, since the machine starts swapping!)
How big is a message sent to each client?
That is, how much memory requires to hold all these messages in memory?
> [...]
P.S.:
your mail user agent does not wraps long lines; can you please fix it?
Manlio Perillo
More information about the Twisted-Python
mailing list