[Twisted-Python] MemoryError in twisted.internet.abstract.FileDescriptor

Glyph glyph at twistedmatrix.com
Thu Dec 15 06:52:59 EST 2011


On Dec 14, 2011, at 7:38 PM, Augusto Mecking Caringi wrote:

> On Wed, Dec 14, 2011 at 9:49 PM, Andrew Bennetts <andrew at bemusement.org> wrote:
>> Augusto Mecking Caringi wrote:
>>>>>> om[3082188716].c
>> […]
>>> str(2917687304
>>> 53120905B 1par 'ype: Test\nDate: 2011-12-14
>>> 16:45:30.995217\nEvent-Subtype: FooBar\nContent-Type:
>>> application/json\n\n{\n ')
>> 
>> So it appears you have 53 megabyte string that's referenced from your TLS
>> connection, presumably in a buffer of data waiting to be sent?  At the moment
>> this memory analysis was captured, do you know how much of this data the client
>> has received?
> 
> Yes... I have a 53MB string referenced in my TLS connection, but *all*
> the data has already been sent.
> 
> At the moment that this memory was captured all the data has been sent.
> 
>> Also, how was this data passed to the transport — in one big 53MB write call, or
>> via many smaller calls, or via a pull producer, or…?
> 
> Via many smaller calls. One call for each event, in a code like that
> (running in a thread):
> 
> def stressTestTask():
>    time.sleep(1)
>    c = provider_reference
>    for i in range(1000000):
>        event = c.createTestEvent()
>        reactor.callFromThread(c.transport.write, event)
> 
> I have two components in my system: the server and the provider, both
> of them written in Python/Twisted. This capture was from a "stress
> test provider" that send hundreds of thousands of events to the server
> without any "sleep". And I captured this memory analysis after the
> server has received all the data.
> 
>> This looks like an HTTP-like protocol, so possibly there's a library between
>> your code and the transport that means you don't know the direct answer to that
>> question.  But whatever details you can give about how you are passing the data
>> you generate to the connection will help locate the problem, I think.
> 
> Yes, this a HTTP-like protocol, but there is no library, the protocol
> was designed by me.
> 
> Thanks.

If everything you've said here is accurate, this is definitely a bug in Twisted.  It would be really valuable if you could boil down your example to be as minimal as possible - ideally a unit test, but failing that, a script which could be trivially run to leak an arbitrary amount of memory would be almost as good.

Does your callFromThread(c.transport.write, ...) code reliably trigger this error when talking to a very simple protocol, like an Echo or Discard protocol?

Thanks a lot for helping us track this down,

-glyph




More information about the Twisted-Python mailing list