[Twisted-web] Finding memory leaks in Nevow/Twisted

Werner Thie wthie at thiengineering.ch
Tue Dec 16 14:39:06 EST 2008


Hi

I was bitten by the same problem and went so far to track down all the 
fishy cycles which easily can be created. Most of the cycles I detected 
were multi-object cycles which were created by me just storing 
references to objects to have them handy. Certain types of cycles cannot 
be broken by the gc, you get a runaway situation either in Python24 or 
Python25. My code now runs without problems for months serving more than 
10k users a day providing a card gaming environment.

Things to check:
- the object hierarchy on the server and the client should be in sync, 
if not detach might not work correctly

- do you use stateful LivePages?
if so, check the toremember list in the request if its growing ad libidum

- do you use guard/sessions?
if so, be careful with the session/mind object and what you store in mind

But without seeing your code it's close to impossible to give you more 
than hints.

I include some code which I wrote to peek into my running server with a 
manhole connection. dumpobjects is loosely based on several Python 
snippets and might not win a contest in programming but it helps to 
track the number of objects of certain types in the running system when 
used on the commandline from inside your server.

HTH, Werner

To create a manhole in the same process you started with your .tac file 
use the following code:

---------------------------------------------------------------
def ManholeFactory(namespace, **passwords):
   realm = manhole_ssh.TerminalRealm()

   def getManhole(_):
     return manhole.Manhole(namespace)

   realm.chainedProtocolFactory.protocolFactory = getManhole
   p = portal.Portal(realm)

p.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**passwords))
   f = manhole_ssh.ConchFactory(p)
   return f

console = ManholeFactory(globals(), admin='admin')
internet.TCPServer(2222, console, 
interface='localhost').setServiceParent(application)
---------------------------------------------------------------

exc = [
   "function",
   "type",
   "list",
   "dict",
   "tuple",
   "wrapper_descriptor",
   "module",
   "method_descriptor",
   "member_descriptor",
   "instancemethod",
   "builtin_function_or_method",
   "frame",
   "classmethod",
   "classmethod_descriptor",
   "_Environ",
   "MemoryError",
   "_Printer",
   "_Helper",
   "getset_descriptor",
   "weakreaf"
]

inc = [
   'YourObject_One',
   'YourObject_Two'
]

prev = {}

def dumpObjects(delta=True, limit=0, include=inc, exclude=[]):
   global prev
   if include != [] and exclude != []:
     print 'cannot use include and exclude at the same time'
     return
   print 'working with:'
   print '   delta: ', delta
   print '   limit: ', limit
   print ' include: ', include
   print ' exclude: ', exclude
   objects = {}
   gc.collect()
   oo = gc.get_objects()
   for o in oo:
     if getattr(o, "__class__", None):
       name = o.__class__.__name__
       if ((exclude == [] and include == [])       or \
           (exclude != [] and name not in exclude) or \
           (include != [] and name in include)):
         objects[name] = objects.get(name, 0) + 1
##    if more:
##      print o
   pk = prev.keys()
   pk.sort()
   names = objects.keys()
   names.sort()
   for name in names:
     if limit == 0 or objects[name] > limit:
       if not prev.has_key(name):
         prev[name] = objects[name]
       dt = objects[name] - prev[name]
       if delta or dt != 0:
         print '%0.6d -- %0.6d -- ' % (dt, objects[name]),  name
       prev[name] = objects[name]







Abdul-Wahid Paterson wrote:
> Hi,
> 
> I am using Python 2.5.
> 
> I will try to use guppy as suggested in another post to so if it will 
> help. Otherwise yes, I will try to reduce it to the smallest possible 
> code to demonstrate the leak. At the moment the code is a bit complex so 
> is hard to track down.
> 
> Thanks for the help.
> 
> AW
> 
> 
> 
> On Tue, Dec 16, 2008 at 6:23 PM, Phil Christensen <phil at bubblehouse.org 
> <mailto:phil at bubblehouse.org>> wrote:
> 
>     On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote:
> 
>         I have managed to write my nevow/twisted site. I am using a lot
>         of Athena elements on my site and they are all working well from
>         a functional point of view. However, the Athena elements are
>         eating up RAM and as I leave things running the Twisted process
>         slowly consumes more and more RAM until the process dies.
> 
> 
>     The first thing would be to check that you're running at least
>     Python 2.5. Previous versions have garbage collection problems on
>     long-running apps like a Twisted web app.
> 
>     Athena-based apps should not inherently leak memory, but adding
>     leaks of your own creation isn't too hard ;-)
> 
>     If you are running 2.5 or later, you should try to reduce your app
>     to the smallest possible test version that demonstrates the leak,
>     and someone here can probably be of more help.
> 
>     -phil
> 
>     _______________________________________________
>     Twisted-web mailing list
>     Twisted-web at twistedmatrix.com <mailto:Twisted-web at twistedmatrix.com>
>     http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web



More information about the Twisted-web mailing list