[Twisted-Python] debugging a memory leak
Maarten ter Huurne
maarten at treewalker.org
Mon Feb 22 21:24:24 EST 2010
On Tuesday 23 February 2010, Alec Matusis wrote:
> When I start the process, both python object sizes and their counts rise
> proportionally to the numbers of reconnected clients, and then they
> stabilize after all clients have reconnected.
> At that moment, the "external" RSS process size is about 260MB. The
> "internal size" of all python objects reported by Heapy is about 150MB.
> After two days, the internal sizes/counts stay the same, but the external
> size grows to 1500MB.
> Python object counts/total sizes are measured from the manhole.
> Is this sufficient to conclude that this is a C memory leak in one of the
> external modules or in the Python interpreter itself?
In general, there are other reasons why heap size and RSS size do not match:
1. pages are empty but not returned to the OS
2. pages cannot be returned to the OS because they are not completely empty
It seems Python has different allocators for small and large objects:
Assuming Python uses malloc for all its allocations (does it?), it is the
malloc implementation that determines whether empty pages are returned to
the OS. Under Linux with glibc (your system?), empty pages are returned, so
there reason 1 does not apply.
Depending on the allocation behaviour of Python, the pages may not be empty
though, so reason 2 is a likely suspect.
Python extensions written in C could also leak or fragment memory. Are you
using any extensions that are not pure Python?
More information about the Twisted-Python