[Twisted-Python] strategies for tracking down memory issues

[Glyph Lefkowitz]

On Jan 17, 2014, at 12:43 PM, Jonathan Vanasco <twisted-python at 2xlp.com> wrote:

> 
> some recent changes to a very-happy twisted daemon have resulted in a process that grows in memory until it crashes the box.  boo!
> 
> looking through the code and logs, i'm wondering if i''ve coded things in such a way that defferds or deferrd lists are somehow not getting cleaned up if an unhandled exception occurs. 
> 
> i've been looking through all my former notes and some questions on stack overflow, and I've seen a lot of info on using heapy and other tools to find issues on a function-by-function basis.  
> 
> i'm wondering if anyone has experience in simply monitoring the lifecycle of deferreds ?

First off, just manhole in and inspect gc.garbage :-).

Second, I've had pretty good luck with <http://www.aminus.net/wiki/Dowser>, although it would be nice if it were a little easier to set up without writing a weird little custom code hook and having to decide where to put it.

However, you can sort reachable objects by count and type, so you can see which of your classes or Twisted's classes are hanging around in memory and that will usually very quickly give you an idea of where the issue might be.

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140117/c035de1b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4124 bytes
Desc: not available
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140117/c035de1b/attachment.bin>