[Twisted-Python] Degrading under load

Yitzchak Gale gale at sefer.org
Sat Mar 11 17:31:35 EST 2006

Itamar Shtull-Trauring wrote:
> Yitzchak Gale wrote:

>> But I think I'll go with B [two processes] anyway.
>> It is neater in that it separates performance under load
>> from the architechture of the high-level processing.
>> With A [simple twisted], we will always have to worry about
>> dividing the XML stuff into small enough pieces
>> to let the event loop in often enough.

> You could also use a thread pool for the XML processing. This will
> certainly help make the event loop stay more responsive without breaking
> up work manually.

I am afraid that a thread pool will not play nicely with
our current monitoring framework.

> However, if the heavy lifting in the code is pure
> Python

It probably will be.

> you won't be able to take advantage of multiple CPUs, because of
> the global interpreter lock.

OK, that is also an issue.

> Processes also have the benefit that you
> can dispatch them to multiple machines.



More information about the Twisted-Python mailing list