[Twisted-Python] Advise for heavy concurrency

[Alec Matusis]

> What causes the CPU overload in such a case

It looks like the for loop that loops over connections to send the message is the main source of the load. The biggest problem is not the CPU however. When the process takes more than 70% of the CPU (as displayed by top), python seems to start skipping garbage collection, and the memory size of the process just keeps growing. We have no control over this.

> I doubt I'll be running 4 heavy cpu-bound processes on a 4-core
> system

Running 4 servers on a 4-core box seems fine, it leaves plenty of resources for system tasks and logging, as long as as long as they are not at 99% CPU each, but I doubt you will get them to run at 99% because of the said memory problem.

> -----Original Message-----
> From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-
> bounces at twistedmatrix.com] On Behalf Of Atilla
> Sent: Thursday, November 29, 2007 10:47 AM
> To: Twisted general discussion
> Subject: Re: [Twisted-Python] Advise for heavy concurrency
> 
> I doubt I'll be running 4 heavy cpu-bound processes on a 4-core
> system, I'd prefer to leave 1 core "free" for all the rest of
> monitoring/logging tasks.
> 
> What causes the CPU overload in such a case? Currently I keep a custom
> many-to-many like structure, which has O(1) access time and need to
> forward the same message to N clients that are returned from that
> structure, for each incoming packet, at worst. I could potentially
> pre-pack the exact network message to avoid generating it for each of
> them, which at this point is the largest overhead. Although I'm not
> sure how much I can save like that.
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python