[Twisted-Python] strategies for tracking down memory issues

Werner Thie werner at thieprojects.ch
Fri Jan 17 14:41:59 MST 2014


On 1/17/14 10:43 AM, Jonathan Vanasco wrote:
>
> some recent changes to a very-happy twisted daemon have resulted in a process that grows in memory until it crashes the box.  boo!
>
> looking through the code and logs, i'm wondering if i''ve coded things in such a way that defferds or deferrd lists are somehow not getting cleaned up if an unhandled exception occurs.
>
> i've been looking through all my former notes and some questions on stack overflow, and I've seen a lot of info on using heapy and other tools to find issues on a function-by-function basis.
>
> i'm wondering if anyone has experience in simply monitoring the lifecycle of deferreds ?

Hi

A few years ago I ran into the problem of non collectable cycles being 
produced by keeping references around in objects locally.

Please find attached a snippet of code I used over and over again for 
hunting down such cycles. It's for sure not a simple cure, but with a 
manhole service being run in your process of question, it gives you the 
telltale signs your looking for.

Use it like

from dumpObjects import dumpObjects

dumpObjects()

Various include/exclude sets should help you to narrow the search in 
consecutive runs.

As for the lifecycle of deferreds I never ever had a problem with 
deferreds not being cleaned up, it has always been me who produced those 
non collectable cycles, usually under the false assumption, that it was 
safe to keep a ref handy.

Werner

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dumpObjects.py
Type: text/x-python
Size: 1935 bytes
Desc: not available
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140117/b3d0ad11/attachment.py>


More information about the Twisted-Python mailing list