[Twisted-Python] Twisted Memory Leaks & Epoll

[Jean-Paul Calderone]

On Tue, 1 May 2007 14:06:43 +0100, Matthew Glubb <matt at zgroupplc.com> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Dear All,
>
>I have been benchmarking my twistd application (twisted.web.server)  and I 
>am seeing increasing memory usage that is never freed. Before I  delve into 
>GC and valgrind, I am interested in learning whether there  are any 
>fundamental no-nos under twisted that might result in a leak.  I am 
>relatively new to twisted and its entirely possible that I am  doing 
>something wrong.

All of the typical Python caveats apply.  There's little that's specific
to Twisted here, aside from general Python rules applied to Twisted objects
and APIs.

>
>I am also interested in learning what the stability of the epoll  reactor 
>is. I have come across a few (old) posts that pointed to it  being 
>incomplete. Would the epoll reactor be a possible cause of a  memory leak? 
>Leak aside, I have benchmarked my app up to 5000  simultaneous connections 
>with no obvious drop in performance.  Switching to poll produces *terrible* 
>results.

The epoll reactor is new in the latest release of Twisted.  It may well have
bugs. :)  It is capable of passing the full Twisted test suite (3100+ tests),
though, which at least says something.

>
>For the purposes of diagnosing the cause of the leak (C extension,  python, 
>twisted, or my code) I have removed external library  dependencies and I 
>have also swapped out reactors. The leak persists.
>
>Does anyone have any good tips for diagnosing leaks under twisted?  For what 
>its worth, I am conducting my tests under an AMD 64bit  architecture.

Having a manhole in a running server is pretty useful.  Once you have that,
you can poke around with the gc module, or maybe something like pysizer, or
things like http://twistedmatrix.com/users/spiv/countrefs.py or
http://twistedmatrix.com/trac/browser/sandbox/exarkun/instcount.py

Jean-Paul