[Twisted-Python] MemoryError in twisted.internet.abstract.FileDescriptor

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Sun Dec 11 14:11:17 EST 2011


On 10 Dec, 05:25 pm, augustocaringi at gmail.com wrote:
>On Fri, Dec 9, 2011 at 7:31 PM, Itamar Turner-Trauring
><itamar at itamarst.org> wrote:
>>>    What's the best solution? Apply the patch attached on this 
>>>ticket,
>>>moving to a producer/consumer approach, or any other idea?
>>
>>The patch will just delay the problem... you're creating a huge number 
>>of
>>strings, faster than the transport can write them out. The solution is
>>indeed to use the consumer API to pause creation of more data until 
>>the
>>transport clears its buffers.
>
>Right, I will try with the consumer API.
>
>But I have one last question:
>
>In my previous example, the memory usage grows until a MemoryError 
>exception.
>
>But other scenario is when my "event send loop" iterates a high number
>of times (but not enough to raise a exception) and then stops.
>
>I expected that when the loop ends, all the strings would be flushed
>and as a consequence, the memory usage of the process would return to
>a normal level. But this does not happen... It's normal?

Data may or may not be put onto the network as you are directing a 
transport to write it.  It's up to the particular transport 
implementation to decide on buffering logic, including logic about 
whether data is sent immediately when a write() call is, or only later 
after control returns to the event loop.

As of Twisted 11.1, the posix-based reactor implementations all buffer 
data until control is returned to the event loop.  This has been the 
case for some time, but not _all_ time, and it may change in the future.

Jean-Paul



More information about the Twisted-Python mailing list