[Twisted-Python] Advise for heavy concurrency
Manlio Perillo
manlio_perillo at libero.it
Sat Dec 1 05:05:59 EST 2007
Alec Matusis ha scritto:
>> What causes the CPU overload in such a case
>
> It looks like the for loop that loops over connections to send the message
> is the main source of the load. The biggest problem is not the CPU however.
> When the process takes more than 70% of the CPU (as displayed by top), python seems to start
> skipping garbage collection, and the memory size of the process just keeps growing. We have no control over this.
>
You should break the loop into small parts.
If you send a message to a lot of clients, then Twisted needs to keep
all the buffers in memory (since your loop is blocking the reactor and
Twisted is unable to flush the buffers).
You see the memory growing because the Python interpreter will not
release the memory to the operating system (however the memory should
not grow indefinitely)
> [...]
Manlio Perillo
More information about the Twisted-Python
mailing list