[Twisted-Python] help with refcounts and memleaks

Andrea Arcangeli andrea at cpushare.com
Mon Dec 26 22:27:24 EST 2005


On Tue, Dec 27, 2005 at 01:47:42PM +1100, Christopher Armstrong wrote:
>  - gc.collect() is the equivalent of what's run occassionally by

IMHO the "occasionally" is the only wrong thing of the whole story. It
must not be "occasionally", it must be "occasionally _or_ when the task
is growing to an insanse size". The total amount of anonymous memory
allocated by the interpreter must be tracked in O(1) and a collect()
should have been invokved at least every time the amount of memory
doubled.

The reason it took me so long before I could suspect the gc, is that
coming from a vm kernel background, I couldn't even dream that after the
task grown up to >1G and the system was into swap, the python gc didn't
even yet try to prune all potentially freeable objects. The gc should
definitely be in function of "size" too, and currently it's not. There
is definitely room for improvements in the gc by adding heuristics in
function of "size of anonymous memory allocated", and it doesn't seem
difficult to add it, nor it should impact performance since the
deep gc.collect() (the only costly thing) would very rarely be invoked
more frequently.




More information about the Twisted-Python mailing list