[Twisted-Python] debugging a memory leak

[Alec Matusis]

Hi Maarten,

Your link
http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-
a-large-object.htm 
seems to suggest that even though the interpreter does not release memory
back to the OS, it can be re-used by the interpreter.
If this was our problem, I'd expect the memory to be set by the highest
usage, as opposed to it constantly leaking: in my case, the load is
virtually constant, but the memory still leaks over time.

The environment is Linux 2.6.24 x86-64, the extensions used are MySQLdb,
pyCrypto (latest stable releases for both).

> -----Original Message-----
> From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-
> bounces at twistedmatrix.com] On Behalf Of Maarten ter Huurne
> Sent: Monday, February 22, 2010 6:24 PM
> To: Twisted general discussion
> Subject: Re: [Twisted-Python] debugging a memory leak
> 
> On Tuesday 23 February 2010, Alec Matusis wrote:
> 
> > When I start the process, both python object sizes and their counts rise
> > proportionally to the numbers of reconnected clients, and then they
> > stabilize after all clients have reconnected.
> > At that moment, the "external" RSS process size is about 260MB. The
> > "internal size" of all python objects reported by Heapy is about 150MB.
> > After two days, the internal sizes/counts stay the same, but the
external
> > size grows to 1500MB.
> >
> > Python object counts/total sizes are measured from the manhole.
> > Is this sufficient to conclude that this is a C memory leak in one of
the
> > external modules or in the Python interpreter itself?
> 
> In general, there are other reasons why heap size and RSS size do not
match:
> 1. pages are empty but not returned to the OS
> 2. pages cannot be returned to the OS because they are not completely
empty
> 
> It seems Python has different allocators for small and large objects:
> http://www.mail-archive.com/python-list@python.org/msg256116.html
> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-
> delete-
> a-large-object.htm
> 
> Assuming Python uses malloc for all its allocations (does it?), it is the
> malloc implementation that determines whether empty pages are returned to
> the OS. Under Linux with glibc (your system?), empty pages are returned,
so
> there reason 1 does not apply.
> 
> Depending on the allocation behaviour of Python, the pages may not be
> empty
> though, so reason 2 is a likely suspect.
> 
> Python extensions written in C could also leak or fragment memory. Are you
> using any extensions that are not pure Python?
> 
> Bye,
> 		Maarten
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python