[Twisted-Python] MemoryError in twisted.internet.abstract.FileDescriptor

Wed Dec 14 16:52:56 MST 2011

On 08:34 pm, augustocaringi at gmail.com wrote:
>On Sun, Dec 11, 2011 at 5:11 PM,  <exarkun at twistedmatrix.com> wrote:
>>On 10 Dec, 05:25 pm, augustocaringi at gmail.com wrote:
>>>
>>>On Fri, Dec 9, 2011 at 7:31 PM, Itamar Turner-Trauring
>>><itamar at itamarst.org> wrote:
>>>>>
>>>>>    What's the best solution? Apply the patch attached on this 
>>>>>ticket,
>>>>>moving to a producer/consumer approach, or any other idea?
>>>>
>>>>
>>>>The patch will just delay the problem... you're creating a huge 
>>>>number of
>>>>strings, faster than the transport can write them out. The solution 
>>>>is
>>>>indeed to use the consumer API to pause creation of more data until 
>>>>the
>>>>transport clears its buffers.
>>>
>>>
>>>Right, I will try with the consumer API.
>>>
>>>But I have one last question:
>>>
>>>In my previous example, the memory usage grows until a MemoryError
>>>exception.
>>>
>>>But other scenario is when my "event send loop" iterates a high 
>>>number
>>>of times (but not enough to raise a exception) and then stops.
>>>
>>>I expected that when the loop ends, all the strings would be flushed
>>>and as a consequence, the memory usage of the process would return to
>>>a normal level. But this does not happen... It's normal?
>>
>>
>>Data may or may not be put onto the network as you are directing a 
>>transport
>>to write it.  It's up to the particular transport implementation to 
>>decide
>>on buffering logic, including logic about whether data is sent 
>>immediately
>>when a write() call is, or only later after control returns to the 
>>event
>>loop.
>>
>>As of Twisted 11.1, the posix-based reactor implementations all buffer 
>>data
>>until control is returned to the event loop.  This has been the case 
>>for
>>some time, but not _all_ time, and it may change in the future.
>>
>>Jean-Paul
>
>Hi!
>
>I understand...
>
>But I'm not convinced.
>
>I found this post on Stackoverflow:
>http://stackoverflow.com/questions/4078084/memory-leak-in-python- 
>twisted-where-is-it
>
>The problem that I am facing is the same.
>
>Under heavy load my Twisted based server "leaks memory". Under normal
>load its all ok.
>
>Explaining better: Under heavy load the memory usage grows, but when
>the load returns to a normal level, the memory usage remains high.

This is basically typical.  I won't guarantee that your program isn't 
*leaking* memory, but it sounds like you're just observing the "high 
water mark" behavior of the CPython runtime (and to a lesser extend of 
glibc).  Allocate memory is frequently not released to the platform when 
it is no longer in use, because fragmentation has happened that makes 
this impractical.

Can you clarify if this is what you're seeing, or if you do see memory 
usage continue to rise without bound (under a bounded load)?

Jean-Paul
>I forgot to say in the previous emails, but the communication is done 
>over SSL.
>
>I'm not a Python specialist, I am a C engineer... But I'm trying to
>find out where is the problem with tools like Heapy, meliae, etc.
>
>Meliae gave me this report:
>
>Total 62008 objects, 126 types, Total size = 86.2MiB (90437121 bytes)
>Index   Count   %      Size   % Cum     Max Kind
>     0   39061  62  87827505  97  9753120905 str
>     1    1682   2    896912   0  98  393352 dict
>     2   10067  16    382028   0  98     224 tuple
>     3     164   0    246656   0  98    6304 module
>     4     422   0    183992   0  99     436 type
>     5    2504   4    170272   0  99      68 code
>
>As expected... The problem are the strings that are never freed.
>
>So I tried to tracking this strings, and this let me to this:
>>>>om[3082188716]
>TLSConnection(3082188716 548B 35refs 38par)
>>>>om[3082188716].c
>[str(3082713312 39B 8par '_tempDataBuffer'), list(3082225612 112704B
>26680refs 1par), str(3086791008 30B 54par 'fileno'),
>instancemethod(3084243748 36B 3refs 1par), str(3084439680 32B 58par
>'protocol'), Provider(3082216172 552B 23refs 4par), str(3084439648 31B
>100par 'reactor'), SelectReactor(3082464620 548B 43refs 21par),
>str(3082712592 37B 6par '_userWantRead'), bool(40623276 12B 94par
>'True'), str(3082713272 34B 8par 'dataBuffer'), str(2917687304
>53120905B 1par 'ype: Test\nDate: 2011-12-14
>16:45:30.995217\nEvent-Subtype: FooBar\nContent-Type:
>application/json\n\n{\n '), str(3082831376 36B 6par '_tempDataLen'),
>int(165862528 12B 1par 33109881), str(3082714752 35B 6par
>'realAddress'), tuple(3082165324 32B 2refs 1par), str(3084479288 33B
>29par 'connector'), Connector(3082188684 552B 21refs 1par),
>str(3082724352 30B 9par 'logstr'), str(3082181232 39B 1par
>'Provider,client'), str(3082723680 27B 5par 'TLS'), int(164303008 12B
>100par 1), str(3084439744 30B 80par 'socket'), Connection(3082208356
>40B 3refs 1par), str(3084506080 33B 33par 'connected'), int(164303008
>12B 100par 1), str(3087080928 30B 10par 'offset'), int(167915152 12B
>1par 11436032), str(3082712632 38B 6par '_userWantWrite'),
>bool(40623276 12B 94par 'True'), str(3082685960 34B 8par
>'ctxFactory'), ClientContextFactory(3082188652 168B 1refs 2par),
>str(3086955200 28B 38par 'addr'), tuple(3082649516 32B 2refs 1par),
><ex-reference>(0 0B)]
>
>Thanks and sorry for the insistence.
>
>--
>Augusto Mecking Caringi
>
>_______________________________________________
>Twisted-Python mailing list
>Twisted-Python at twistedmatrix.com
>http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python