[Twisted-Python] help with refcounts and memleaks

Andrea Arcangeli andrea at cpushare.com
Mon Dec 26 20:20:07 MST 2005


On Tue, Dec 27, 2005 at 03:28:30AM +0100, Jesus Cea wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Wow, Andrea Arcangeli, of Linux Kernel fame. What an honor :).
> 
> > +		import gc
> > +		gc.collect()
> 
> Haha. I was just writing you about manual garbage collector invocations.

;) Thanks! I couldn't even think that it could have been the garbage
collector because the thing was >1G large. I definitely expected any
sane garbage collector to kick in at least once every time the task was
doubling in size and certainly before the thing could grow to the
gigabytes. Perhaps this is what should really happen. Every time the
task doubles in size we could force a gc.collect(). The growth rate that
should trigger a collect() could be a threshold tunable too (default
None to be backwards compatible even though I think None is not a
safe/sane default).

> Manual collections can be costly, nevertheless. Since the manual
> collector is only needed to free cyclic references, I guess that Twisted
> people would rather prefer a patch to break cyclic references in Twisted
> code. For example, using "__del__" methods or "weakref".
> 
> You can easily see what objects and references keep the cycles alive
> using "gc.garbage" and "gc.get_referrers()". Usually a single
> "a.value=None" can do miracles :-). Weakref can be very very helpful, also.

a.value=None + weakrefs is what did the miracles indeed, but I did it in my
code not in twisted code. twisted code has no cyclic references. All
cyclic references were in my code sitting on top of twisted. It's me
implementing the protocol.

The a.value=None made the code a mess, the weakref complicated things
too (especially because I had to hash one of those weak values, so I had
to use weakref.ref and I had to change lots of code to add "()"). So I'm
soooo glad to have backed out all those changes because they were
unnecessary ;). Plus a.value = None is similar to freeing memory in a
normal language without garbage collection, not very different from
free(a.value). So I certainly prefer to use the more advanced features
of the garbage collector, even if I have to invoke it by hand... ;)




More information about the Twisted-Python mailing list