[Twisted-Python] Advise for heavy concurrency

[Alec Matusis]

> How big is a message sent to each client?

Each message is about 40 bytes.
A typical loop to send message is 1000 clients, 
so only 40KB to hold the message for all clients.

> -----Original Message-----
> From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-
> bounces at twistedmatrix.com] On Behalf Of Manlio Perillo
> Sent: Saturday, December 01, 2007 3:55 AM
> To: Twisted general discussion
> Subject: Re: [Twisted-Python] Advise for heavy concurrency
> 
> Alec Matusis ha scritto:
> >> You should break the loop into small parts.
> >
> > Can you explain this?
> > Do you mean, use reactor.callLater to process say 100 messages at a
> time? Or maybe use deferToThread for the whole loop?
> >
> 
> There is also twisted.internet.task.cooperate, but I have never used
> it.
> 
> >> You see the memory growing because the Python interpreter will not
> >> release the memory to the operating system (however the memory
> should
> >> not grow indefinitely)
> >
> > The memory is actually growing indefinitely.
> > We run 4 identical servers on one machine. The load is not balanced
> very symmetrically.
> > So one server can have 4500 connections, while another has 6000.
> > There is a certain threshold on the number of connections, when a
> server starts to leak memory uncontrollably,
> > and it's not set by the highest usage. For example, a server with
> 4500 connections remains at 100mb RSS for days,
> > while a server that reaches 6000 connections leaks memory to 2gb in
> 10 hrs
> > (I did not try beyond that, since the machine starts swapping!)
> 
> 
> How big is a message sent to each client?
> That is, how much memory requires to hold all these messages in memory?
> 
>  > [...]
> 
> 
> 
> P.S.:
> your mail user agent does not wraps long lines; can you please fix it?
> 
> 
> 
> 
> 
> Manlio Perillo
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python