[Twisted-Python] Advise for heavy concurrency

[Alec Matusis]

> You should break the loop into small parts.

Can you explain this?
Do you mean, use reactor.callLater to process say 100 messages at a time? Or maybe use deferToThread for the whole loop?

> You see the memory growing because the Python interpreter will not
> release the memory to the operating system (however the memory should
> not grow indefinitely)

The memory is actually growing indefinitely.
We run 4 identical servers on one machine. The load is not balanced very symmetrically. So one server can have 4500 connections, while another has 6000. There is a certain threshold on the number of connections, when a server starts to leak memory uncontrollably, and it's not set by the highest usage. For example, a server with 4500 connections remains at 100mb RSS for days, while a server that reaches 6000 connections leaks memory to 2gb in 10 hrs (I did not try beyond that, since the machine starts swapping!)  

> -----Original Message-----
> From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-
> bounces at twistedmatrix.com] On Behalf Of Manlio Perillo
> Sent: Saturday, December 01, 2007 2:06 AM
> To: Twisted general discussion
> Subject: Re: [Twisted-Python] Advise for heavy concurrency
> 
> Alec Matusis ha scritto:
> >> What causes the CPU overload in such a case
> >
> > It looks like the for loop that loops over connections to send the
> message
> > is the main source of the load. The biggest problem is not the CPU
> however.
> > When the process takes more than 70% of the CPU (as displayed by
> top), python seems to start
> > skipping garbage collection, and the memory size of the process just
> keeps growing. We have no control over this.
> >
> 
> You should break the loop into small parts.
> 
> If you send a message to a lot of clients, then Twisted needs to keep
> all the buffers in memory (since your loop is blocking the reactor and
> Twisted is unable to flush the buffers).
> 
> You see the memory growing because the Python interpreter will not
> release the memory to the operating system (however the memory should
> not grow indefinitely)
> 
> 
>  > [...]
> 
> 
> Manlio Perillo
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python