[Twisted-Python] debugging a memory leak

Werner Thie wthie at thiengineering.ch
Tue Feb 23 00:39:01 MST 2010


Hi

Assuming that if memory not released to the OS can be reused by the 
interpreter because of a suballocation system used in the interpreter 
should eventually lead to a leveling out of the overall memory usage 
over time, that's what I observe with our processes (sitting at several 
100 MB per process). We are using external C libraries which do lots of 
malloc/free and one of the bigger sources of pain is indeed to bring 
such a library to a point where its clean not only by freeing all memory 
allocated in every circumstance but also Python refcounting wise. I 
usually go thru all the motions to build up a complete debug chain for 
all modules involved in a project and write a test bed to proof clean 
and proper implementation.

So if your using C/C++ based modules in your project I would mark them 
as highly suspicious to be responsible for leaks until proven otherwise.

Not to bother you with numbers but I usually allocate about 30% of 
overall project time to bring a server into a production ready state, 
meaning uptimes of months/years, no fishy feelings, no performance 
oscillations, predictable caving and recuperating when overloaded, just 
all the things you have to tick to sign off a project as completed, 
meaning you don't have to do daily 'tire kicking' maintenance and 
periodic reboots.

Werner

Alec Matusis wrote:
> Hi Maarten,
> 
> Your link
> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-
> a-large-object.htm 
> seems to suggest that even though the interpreter does not release memory
> back to the OS, it can be re-used by the interpreter.
> If this was our problem, I'd expect the memory to be set by the highest
> usage, as opposed to it constantly leaking: in my case, the load is
> virtually constant, but the memory still leaks over time.
> 
> The environment is Linux 2.6.24 x86-64, the extensions used are MySQLdb,
> pyCrypto (latest stable releases for both).
> 
>> -----Original Message-----
>> From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-
>> bounces at twistedmatrix.com] On Behalf Of Maarten ter Huurne
>> Sent: Monday, February 22, 2010 6:24 PM
>> To: Twisted general discussion
>> Subject: Re: [Twisted-Python] debugging a memory leak
>>
>> On Tuesday 23 February 2010, Alec Matusis wrote:
>>
>>> When I start the process, both python object sizes and their counts rise
>>> proportionally to the numbers of reconnected clients, and then they
>>> stabilize after all clients have reconnected.
>>> At that moment, the "external" RSS process size is about 260MB. The
>>> "internal size" of all python objects reported by Heapy is about 150MB.
>>> After two days, the internal sizes/counts stay the same, but the
> external
>>> size grows to 1500MB.
>>>
>>> Python object counts/total sizes are measured from the manhole.
>>> Is this sufficient to conclude that this is a C memory leak in one of
> the
>>> external modules or in the Python interpreter itself?
>> In general, there are other reasons why heap size and RSS size do not
> match:
>> 1. pages are empty but not returned to the OS
>> 2. pages cannot be returned to the OS because they are not completely
> empty
>> It seems Python has different allocators for small and large objects:
>> http://www.mail-archive.com/python-list@python.org/msg256116.html
>> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-
>> delete-
>> a-large-object.htm
>>
>> Assuming Python uses malloc for all its allocations (does it?), it is the
>> malloc implementation that determines whether empty pages are returned to
>> the OS. Under Linux with glibc (your system?), empty pages are returned,
> so
>> there reason 1 does not apply.
>>
>> Depending on the allocation behaviour of Python, the pages may not be
>> empty
>> though, so reason 2 is a likely suspect.
>>
>> Python extensions written in C could also leak or fragment memory. Are you
>> using any extensions that are not pure Python?
>>
>> Bye,
>> 		Maarten
>>
>> _______________________________________________
>> Twisted-Python mailing list
>> Twisted-Python at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




More information about the Twisted-Python mailing list