[Twisted-Python] profiling twisted
Maarten ter Huurne
maarten at treewalker.org
Wed Jun 27 13:04:18 EDT 2007
On Wednesday 27 June 2007, Markus Schiltknecht wrote:
> Now, I'm wondering if there's a profiling tool, which can quickly tell
> me what events are blocking my server. I have even deferred the file I/O
> to separate threads, as I've suspected them to make the server behave
> poorly when under load. But there still seems to be something
> blocking... What do you use to find the callbacks which block (for too
Does your server process use a lot of memory?
I had problems with a random simple request taking a long time every now and
then and it turned out to be the Python garbage collector. The collection
itself can take some time if you have a lot of objects. If some of the Python
process ends up in the swap instead of main memory, garbage collection will
block until it is swapped back in, which can slow it down dramatically.
Using multiple threads does not give you any performance advantage in this
case: during garbage collection the entire Python VM is frozen.
One thing you could try is:
in the startup code of your server.
Note that this does not turn off all memory recycling; it only turns off the
mark-and-sweep part of the garbage collector. Objects which have a zero
reference count are still collected.
More information about the Twisted-Python