[Twisted-Python] Degrading under load
itamar at itamarst.org
Fri Mar 10 11:50:10 EST 2006
On Fri, 2006-03-10 at 02:02 +0200, Yitzchak Gale wrote:
> But I think I'll go with B anyway. It is neater in
> that it separates performance under load
> from the architechture of the high-level processing.
> With A, we will always have to worry about
> dividing the XML stuff into small enough pieces
> to let the event loop in often enough.
You could also use a thread pool for the XML processing. This will
certainly help make the event loop stay more responsive without breaking
up work manually. However, if the heavy lifting in the code is pure
Python you won't be able to take advantage of multiple CPUs, because of
the global interpreter lock. Processes also have the benefit that you
can dispatch them to multiple machines.
More information about the Twisted-Python