[Twisted-web] Finding memory leaks in Nevow/Twisted

[Abdul-Wahid Paterson]

Werner,

Thanks for your post. Is very useful.

I am using Guard so that might be one place to start looking although I
personally didn't develop that part of the code. (Which kinda make me even
more suspicious).

The Manhole stuff looks interesting too. I will give that a go if I can't
find anything by disabling the Guard code.

Thanks again.

AW

On Tue, Dec 16, 2008 at 10:39 PM, Werner Thie <wthie at thiengineering.ch>wrot=
e:

> Hi
>
> I was bitten by the same problem and went so far to track down all the
> fishy cycles which easily can be created. Most of the cycles I detected w=
ere
> multi-object cycles which were created by me just storing references to
> objects to have them handy. Certain types of cycles cannot be broken by t=
he
> gc, you get a runaway situation either in Python24 or Python25. My code n=
ow
> runs without problems for months serving more than 10k users a day provid=
ing
> a card gaming environment.
>
> Things to check:
> - the object hierarchy on the server and the client should be in sync, if
> not detach might not work correctly
>
> - do you use stateful LivePages?
> if so, check the toremember list in the request if its growing ad libidum
>
> - do you use guard/sessions?
> if so, be careful with the session/mind object and what you store in mind
>
> But without seeing your code it's close to impossible to give you more th=
an
> hints.
>
> I include some code which I wrote to peek into my running server with a
> manhole connection. dumpobjects is loosely based on several Python snippe=
ts
> and might not win a contest in programming but it helps to track the numb=
er
> of objects of certain types in the running system when used on the
> commandline from inside your server.
>
> HTH, Werner
>
> To create a manhole in the same process you started with your .tac file u=
se
> the following code:
>
> ---------------------------------------------------------------
> def ManholeFactory(namespace, **passwords):
>  realm =3D manhole_ssh.TerminalRealm()
>
>  def getManhole(_):
>    return manhole.Manhole(namespace)
>
>  realm.chainedProtocolFactory.protocolFactory =3D getManhole
>  p =3D portal.Portal(realm)
>
>
> p.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**pass=
words))
>  f =3D manhole_ssh.ConchFactory(p)
>  return f
>
> console =3D ManholeFactory(globals(), admin=3D'admin')
> internet.TCPServer(2222, console,
> interface=3D'localhost').setServiceParent(application)
> ---------------------------------------------------------------
>
> exc =3D [
>  "function",
>  "type",
>  "list",
>  "dict",
>  "tuple",
>  "wrapper_descriptor",
>  "module",
>  "method_descriptor",
>  "member_descriptor",
>  "instancemethod",
>  "builtin_function_or_method",
>  "frame",
>  "classmethod",
>  "classmethod_descriptor",
>  "_Environ",
>  "MemoryError",
>  "_Printer",
>  "_Helper",
>  "getset_descriptor",
>  "weakreaf"
> ]
>
> inc =3D [
>  'YourObject_One',
>  'YourObject_Two'
> ]
>
> prev =3D {}
>
> def dumpObjects(delta=3DTrue, limit=3D0, include=3Dinc, exclude=3D[]):
>  global prev
>  if include !=3D [] and exclude !=3D []:
>    print 'cannot use include and exclude at the same time'
>    return
>  print 'working with:'
>  print '   delta: ', delta
>  print '   limit: ', limit
>  print ' include: ', include
>  print ' exclude: ', exclude
>  objects =3D {}
>  gc.collect()
>  oo =3D gc.get_objects()
>  for o in oo:
>    if getattr(o, "__class__", None):
>      name =3D o.__class__.__name__
>      if ((exclude =3D=3D [] and include =3D=3D [])       or \
>          (exclude !=3D [] and name not in exclude) or \
>          (include !=3D [] and name in include)):
>        objects[name] =3D objects.get(name, 0) + 1
> ##    if more:
> ##      print o
>  pk =3D prev.keys()
>  pk.sort()
>  names =3D objects.keys()
>  names.sort()
>  for name in names:
>    if limit =3D=3D 0 or objects[name] > limit:
>      if not prev.has_key(name):
>        prev[name] =3D objects[name]
>      dt =3D objects[name] - prev[name]
>      if delta or dt !=3D 0:
>        print '%0.6d -- %0.6d -- ' % (dt, objects[name]),  name
>      prev[name] =3D objects[name]
>
>
>
>
>
>
>
> Abdul-Wahid Paterson wrote:
>
>> Hi,
>>
>> I am using Python 2.5.
>>
>> I will try to use guppy as suggested in another post to so if it will
>> help. Otherwise yes, I will try to reduce it to the smallest possible co=
de
>> to demonstrate the leak. At the moment the code is a bit complex so is h=
ard
>> to track down.
>>
>> Thanks for the help.
>>
>> AW
>>
>>
>>
>> On Tue, Dec 16, 2008 at 6:23 PM, Phil Christensen <phil at bubblehouse.org<=
mailto:
>> phil at bubblehouse.org>> wrote:
>>
>>    On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote:
>>
>>        I have managed to write my nevow/twisted site. I am using a lot
>>        of Athena elements on my site and they are all working well from
>>        a functional point of view. However, the Athena elements are
>>        eating up RAM and as I leave things running the Twisted process
>>        slowly consumes more and more RAM until the process dies.
>>
>>
>>    The first thing would be to check that you're running at least
>>    Python 2.5. Previous versions have garbage collection problems on
>>    long-running apps like a Twisted web app.
>>
>>    Athena-based apps should not inherently leak memory, but adding
>>    leaks of your own creation isn't too hard ;-)
>>
>>    If you are running 2.5 or later, you should try to reduce your app
>>    to the smallest possible test version that demonstrates the leak,
>>    and someone here can probably be of more help.
>>
>>    -phil
>>
>>    _______________________________________________
>>    Twisted-web mailing list
>>    Twisted-web at twistedmatrix.com <mailto:Twisted-web at twistedmatrix.com>
>>    http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Twisted-web mailing list
>> Twisted-web at twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
>>
>
> _______________________________________________
> Twisted-web mailing list
> Twisted-web at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-web/attachments/20081217/cd=
d42b56/attachment.htm