[Twisted-Python] help with refcounts and memleaks

Andrea Arcangeli andrea at cpushare.com
Mon Dec 26 20:02:37 EST 2005


On Tue, Dec 27, 2005 at 01:18:33AM +0100, Andrea Arcangeli wrote:
> So, I'll backout all my latest changes, and I'll try to find the real

Now after understanding the __del__ heisenbug in my testcases, and a
1000 lines backout (dropping all weakrefs and hacks I did to try to
remove those pratical memleaks) here my findings on the real life
reconnectingclientfactory memleak that made me look into this in the
first place: It wasn't a memleak in theory, it was only in practice...
gc.collect() seems not invoked frequently enough.

Here it is a ps v of my reconnectingclientfactory with an artificial disconnect
from the server at every second (as said before about 50M were attached to
every new protocol instance).

while :; do ps v |grep [c]pushare-0; sleep 1; done
20113 pts/6    SNl+   0:00      0     2 94453 51492  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:00      0     2 115017 70120  4.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:00      0     2 135513 92528  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:00      0     2 135513 92528  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:00      0     2 135513 92528  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:01      0     2 176497 133512  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:01      0     2 176497 133512  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:01      0     2 176497 133512  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:01      0     2 217481 174496 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:01      0     2 217481 174496 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:01      0     2 217481 174496 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:01      0     2 278957 216252 14.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 94529 51568  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 94529 51568  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:02      0     2 125269 82252  5.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 135513 92584  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 135513 92584  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 135513 92584  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 176497 133580  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 176497 133580  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 176497 133596  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 135513 92584  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 135513 92584  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 135513 92584  6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 176497 133580  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 176497 133580  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:02      0     2 176497 133596  8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 217481 174588 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 217481 174588 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 217481 174596 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 258465 215580 14.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 258465 215580 14.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 258465 215580 14.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 299449 256584 16.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 299449 256584 16.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:03      0     2 299449 256584 16.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:04      0     2 319937 275096 17.8 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:04      0     2 340433 297568 19.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:04      0     2 340433 297568 19.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:04      0     2 340433 297584 19.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:04      0     2 381417 338568 22.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:04      0     2 381417 338568 22.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:04      0     2 381417 338568 22.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:05      0     2 412221 369312 24.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:05      0     2 422469 379560 24.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:04      0     2 381417 338568 22.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:05      0     2 412221 369312 24.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:05      0     2 422469 379560 24.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:05      0     2 422469 379560 24.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:05      0     2 422537 379572 24.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:05      0     2 463457 420556 27.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:05      0     2 463457 420556 27.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:05      0     2 463457 420556 27.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:06      0     2 524933 472712 30.7 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:06      0     2 504441 461548 30.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:06      0     2 504441 461548 30.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:06      0     2 524929 482044 31.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:07      0     2 545425 502540 32.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:07      0     2 545425 502540 32.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:07      0     2 545425 502540 32.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:07      0     2 586413 534096 34.7 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:07      0     2 586409 543532 35.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:07      0     2 586409 543532 35.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:07      0     2 586497 543540 35.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:08      0     2 627417 584524 38.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:08      0     2 627417 584524 38.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:08      0     2 627417 584524 38.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:08      0     2 647905 605016 39.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:08      0     2 668401 625512 40.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:08      0     2 668401 625512 40.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:08      0     2 668401 625512 40.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:09      0     2 709389 663952 43.1 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:09      0     2 709385 666500 43.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:09      0     2 709385 666500 43.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:09      0     2 709385 666500 43.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:09      0     2 740121 697240 45.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:10      0     2 750369 707488 45.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:10      0     2 750369 707488 45.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:10      0     2 750369 707488 45.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:10      0     2 791357 744356 48.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:10      0     2 791353 748248 48.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:10      0     2 791353 748248 48.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:10      0     2 791353 748252 48.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:11      0     2 842581 790452 51.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:11      0     2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:11      0     2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:11      0     2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:11      0     2 863077 819348 53.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:12      0     2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:12      0     2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:11      0     2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:11      0     2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:11      0     2 863077 819348 53.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:12      0     2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:12      0     2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:12      0     2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:12      0     2 904061 860200 55.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    DNl+   0:12      0     2 924549 874220 56.8 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    DNl+   0:12      0     2 934797 883008 57.4 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    DNl+   0:12      0     2 934797 890176 57.8 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:12      0     2 914305 869848 56.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    SNl+   0:12      0     2 914305 869848 56.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    RNl+   0:12      0     2 914305 869848 56.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    DNl+   0:13      0     2 945041 900420 58.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    DNl+   0:13      0     2 965533 916084 59.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
20113 pts/6    DNl+   0:13      0     2 975781 925724 60.1 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py


At the end the system was heavy into swap: huge memelak. But it shouldn't have
happened, I know how to write code not to generate memleaks, the only thing left
were cyclical references and self references, but there was no visibility at all
of the objects that were supposed to be freed.

Now see the right fix that made the original gigantic memleak go away!

Index: cpushare/proto.py
===================================================================
RCS file: /home/andrea/crypto/cvs/cpushare/client/cpushare/cpushare/proto.py,v
retrieving revision 1.62
diff -u -p -r1.62 proto.py
--- cpushare/proto.py	27 Dec 2005 00:43:34 -0000	1.62
+++ cpushare/proto.py	27 Dec 2005 00:50:13 -0000
@@ -206,6 +206,8 @@ class cpushare_factory(ReconnectingClien
 
 	def buildProtocol(self, addr):
 		self.resetDelay()
+		import gc
+		gc.collect()
 		return ReconnectingClientFactory.buildProtocol(self, addr)
 
 	def clientConnectionLost(self, connector, reason):



And now see the output of the same command with the same
reconnectingclientfactory being restarted once every second, but with the above
two liner fix applied.

21219 pts/6    RNl+   0:00      0     2 73773 30924  2.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:00      0     2 94233 51428  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:00      0     2 94233 51428  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:00      0     2 94233 51428  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    RNl+   0:01      0     2 83985 41188  2.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:01      0     2 94233 51436  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:01      0     2 94233 51436  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    RNl+   0:01      0     2 73737 30944  2.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:01      0     2 94233 51440  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:01      0     2 94233 51440  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    RNl+   0:01      0     2 63497 17496  1.1 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:01      0     2 94233 51440  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:01      0     2 94233 51440  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    RNl+   0:02      0     2 53249 10464  0.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:02      0     2 94233 51448  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:02      0     2 94233 51448  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:02      0     2 53249 10468  0.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:02      0     2 94233 51452  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:02      0     2 94233 51452  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    RNl+   0:02      0     2 63497 15368  0.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51476  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51476  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51476  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51476  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51476  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51476  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51480  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51480  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
21219 pts/6    SNl+   0:03      0     2 94233 51480  3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py

Now the RSS of the task never exceeds 50M, this is perfectly correct and
the expected behaviour since the first place.

The memleak disappeared completely. So it wasn't a memleak, nor a bug in my
code, but it was only the lack of garbage collection invocation! Of course
my previous attempts of removing the cyclical references made the
problem go away artificially but it made my code a total mess (similar
to c++ style), because then the refcount of the protocol was forced to
go down to 0 and so the memory was released synchronously.

So the question now, is how frequently is the garbage collection invoked, and
why do I need to invoke it by myself to avoid a gigantic "pratical"
memleaks?

Of course now I'll keep an explicit gc.collect() into the reconnecting
handler. Should I add an explicit timer rearming itself and invoking the
garbage collection periodically in the server too?

I'm really relieved the article dating back to 99 was totally obsolete
since my code has always been written for 2005 python garbage collection ;)

Thanks a lot for the quick help!




More information about the Twisted-Python mailing list