[Twisted-Python] debugging a memory leak

[Maarten ter Huurne]

On Tuesday 23 February 2010, Alec Matusis wrote:

> When I start the process, both python object sizes and their counts rise
> proportionally to the numbers of reconnected clients, and then they
> stabilize after all clients have reconnected.
> At that moment, the "external" RSS process size is about 260MB. The
> "internal size" of all python objects reported by Heapy is about 150MB.
> After two days, the internal sizes/counts stay the same, but the external
> size grows to 1500MB.
> 
> Python object counts/total sizes are measured from the manhole.
> Is this sufficient to conclude that this is a C memory leak in one of the
> external modules or in the Python interpreter itself?

In general, there are other reasons why heap size and RSS size do not match:
1. pages are empty but not returned to the OS
2. pages cannot be returned to the OS because they are not completely empty

It seems Python has different allocators for small and large objects:
http://www.mail-archive.com/python-list@python.org/msg256116.html
http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-
a-large-object.htm

Assuming Python uses malloc for all its allocations (does it?), it is the 
malloc implementation that determines whether empty pages are returned to 
the OS. Under Linux with glibc (your system?), empty pages are returned, so 
there reason 1 does not apply.

Depending on the allocation behaviour of Python, the pages may not be empty 
though, so reason 2 is a likely suspect.

Python extensions written in C could also leak or fragment memory. Are you 
using any extensions that are not pure Python?

Bye,
		Maarten