Memory leaks (was Re: [Twisted-Python] Twisted Memory Leaks & Epoll)

[Itamar Shtull-Trauring]

On Tue, 2007-05-01 at 14:06 +0100, Matthew Glubb wrote:

> I have been benchmarking my twistd application (twisted.web.server)  
> and I am seeing increasing memory usage that is never freed. Before I  
> delve into GC and valgrind, I am interested in learning whether there  
> are any fundamental no-nos under twisted that might result in a leak.  
> I am relatively new to twisted and its entirely possible that I am  
> doing something wrong.

What version Python? Pre-2.5 Python would never free up memory.

> I am also interested in learning what the stability of the epoll  
> reactor is. I have come across a few (old) posts that pointed to it  
> being incomplete. Would the epoll reactor be a possible cause of a  
> memory leak? Leak aside, I have benchmarked my app up to 5000  
> simultaneous connections with no obvious drop in performance.  
> Switching to poll produces *terrible* results.

So far there has been only one report of problems with epoll reactor. If
it's working for you, just use it.

Also:
1. It's not incomplete so far as we know, since it passes all tests.
(http://twistedmatrix.com/buildbot/)
2. You said below you had memory leak with other reactors, so it seems
unlikely that epoll is the main reason your application is leaking
memory.

> Does anyone have any good tips for diagnosing leaks under twisted?  
> For what its worth, I am conducting my tests under an AMD 64bit  
> architecture.

The gc module should let you, with a little work, figure out what the
most frequently used Python objects are in your code.