[Twisted-Python] MemoryError in twisted.internet.abstract.FileDescriptor

Augusto Mecking Caringi augustocaringi at gmail.com
Wed Dec 14 19:38:34 EST 2011


On Wed, Dec 14, 2011 at 9:49 PM, Andrew Bennetts <andrew at bemusement.org> wrote:
> Augusto Mecking Caringi wrote:
>> >>> om[3082188716].c
> […]
>> str(2917687304
>> 53120905B 1par 'ype: Test\nDate: 2011-12-14
>> 16:45:30.995217\nEvent-Subtype: FooBar\nContent-Type:
>> application/json\n\n{\n ')
>
> So it appears you have 53 megabyte string that's referenced from your TLS
> connection, presumably in a buffer of data waiting to be sent?  At the moment
> this memory analysis was captured, do you know how much of this data the client
> has received?

Yes... I have a 53MB string referenced in my TLS connection, but *all*
the data has already been sent.

At the moment that this memory was captured all the data has been sent.

> Also, how was this data passed to the transport — in one big 53MB write call, or
> via many smaller calls, or via a pull producer, or…?

Via many smaller calls. One call for each event, in a code like that
(running in a thread):

def stressTestTask():
    time.sleep(1)
    c = provider_reference
    for i in range(1000000):
        event = c.createTestEvent()
        reactor.callFromThread(c.transport.write, event)

I have two components in my system: the server and the provider, both
of them written in Python/Twisted. This capture was from a "stress
test provider" that send hundreds of thousands of events to the server
without any "sleep". And I captured this memory analysis after the
server has received all the data.

> This looks like an HTTP-like protocol, so possibly there's a library between
> your code and the transport that means you don't know the direct answer to that
> question.  But whatever details you can give about how you are passing the data
> you generate to the connection will help locate the problem, I think.

Yes, this a HTTP-like protocol, but there is no library, the protocol
was designed by me.

Thanks.

-- 
Augusto Mecking Caringi



More information about the Twisted-Python mailing list