[Twisted-Python] Degrading under load
gale at sefer.org
Sat Mar 11 17:31:35 EST 2006
Itamar Shtull-Trauring wrote:
> Yitzchak Gale wrote:
>> But I think I'll go with B [two processes] anyway.
>> It is neater in that it separates performance under load
>> from the architechture of the high-level processing.
>> With A [simple twisted], we will always have to worry about
>> dividing the XML stuff into small enough pieces
>> to let the event loop in often enough.
> You could also use a thread pool for the XML processing. This will
> certainly help make the event loop stay more responsive without breaking
> up work manually.
I am afraid that a thread pool will not play nicely with
our current monitoring framework.
> However, if the heavy lifting in the code is pure
It probably will be.
> you won't be able to take advantage of multiple CPUs, because of
> the global interpreter lock.
OK, that is also an issue.
> Processes also have the benefit that you
> can dispatch them to multiple machines.
More information about the Twisted-Python