[Twisted-Python] MemoryError in twisted.internet.abstract.FileDescriptor

[Augusto Mecking Caringi]

Hi!

    I'm writing a software using Python Twisted with the following architecture:

    [server]<->[providers]

    The providers (that can be thousands) send events to the
centralized server, usually we have a few events per minute for each
provider, but a burst of events can happen.

    So I write a test provider to simulate the sending of millions of
events without any delay, via a thread like that:

    def stressTestTask():
        time.sleep(1)
        c = provider_reference
        for i in range(1000000):
            event = c.createTestEvent()
            reactor.callFromThread(c.transport.write, event)

    Then I started to analyze memory consumption while running this
test code, and I find out that the memory consumption grows to
gigabytes and finally a MemoryError exception raises:

        Traceback (most recent call last):
          File "/usr/lib/python2.5/site-packages/twisted/python/log.py",
line 84, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "/usr/lib/python2.5/site-packages/twisted/python/log.py",
line 69, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "/usr/lib/python2.5/site-packages/twisted/python/context.py",
line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.5/site-packages/twisted/python/context.py",
line 81, in callWithContext
            return func(*args,**kw)
        --- <exception caught here> ---
          File "/usr/lib/python2.5/site-packages/twisted/internet/selectreactor.py",
line 146, in _doReadOrWrite
            why = getattr(selectable, method)()
          File "/usr/lib/python2.5/site-packages/twisted/internet/tcp.py",
line 177, in doWrite
            return Connection.doWrite(self)
          File "/usr/lib/python2.5/site-packages/twisted/internet/tcp.py",
line 428, in doWrite
            result = abstract.FileDescriptor.doWrite(self)
          File "/usr/lib/python2.5/site-packages/twisted/internet/abstract.py",
line 199, in doWrite
            self.dataBuffer = buffer(self.dataBuffer, self.offset) +
"".join(self._tempDataBuffer)
        exceptions.MemoryError:

    After a search, I found this ticket:
http://twistedmatrix.com/trac/ticket/3682

    And I think that this is exact my problem.

    What's the best solution? Apply the patch attached on this ticket,
moving to a producer/consumer approach, or any other idea?

    Thanks!

-- 
Augusto Mecking Caringi