id,summary,reporter,owner,description,type,status,priority,milestone,component,resolution,keywords,cc,branch,branch_author,launchpad_bug
3682,twisted.internet.abstract.FileDescriptor accumulates _tempData and throws MemoryError,gthomas,gthomas,"twisted.internet.abstract.FileDescriptor.doWrite method sends some data, but doesn't care about _tempDataBuffer's size.

In the write method, it tries to pause the producer, but too much ifs are there (self.provider exists and it is a not a StreamProducer)).

So when the producer produces too much data, the 
{{{
#!python
        """".join(self._tempDataBuffer) #line 101
}}}
throws a MemoryError.

My suggestions: 

  1. if self.dataBuffer is small, only just a few chunks should be appended to it from self._tempDataBuffer
  1. the doWrite method should try to slim self._tempDataBuffer at least till self.bufferSize (so too much data just can't accumulate in self.dataBuffer and self._tempDataBuffer).

A possible solution (twisted.internet.abstract.FileDescriptor.doWrite:
{{{
#!python
def doWrite(self):
    """"""Called when data can be written.

    A result that is true (which will be a negative number) implies the
    connection was lost. A false result implies the connection is still
    there; a result of 0 implies no write was done, and a result of None
    indicates that a write was done.
    """"""
    #LOG.debug('%s.doWrite: db=%d tdb=%d'
    #          % (self, len(self.dataBuffer)-self.offset, self._tempDataLen))
    while 1:
        # always try to do something
        # quit when dataBuffer size is acceptably low (see end)
        if len(self.dataBuffer) - self.offset < self.SEND_LIMIT:
            # If there is currently less than SEND_LIMIT bytes left to send
            # in the string, extend it with the array data.
            buf = []
            accum = len(self.dataBuffer)-self.offset
            m = 2 * max(self.SEND_LIMIT, self.bufferSize)
            chunk = None
            while self._tempDataBuffer and accum < m:
                # don't want to join too much data at once
                chunk = self._tempDataBuffer.pop(0)
                buf.append(chunk)
                n = len(chunk)
                accum += n
                self._tempDataLen -= n
            #LOG.debug('%s.doWrite: accum=%d tdb=%d' % (self, accum, self._tempDataLen,))
            del chunk, accum, m
            self.dataBuffer = (buffer(self.dataBuffer, self.offset) + """".join(buf))
            del buf
            self.offset = 0

        # Send as much data as you can.
        if self.offset:
            l = self.writeSomeData(buffer(self.dataBuffer, self.offset))
        else:
            l = self.writeSomeData(self.dataBuffer)

        # There is no writeSomeData implementation in Twisted which returns
        # 0, but the documentation for writeSomeData used to claim negative
        # integers meant connection lost.  Keep supporting this here,
        # although it may be worth deprecating and removing at some point.
        if l < 0 or isinstance(l, Exception):
            return l
        if l == 0 and self.dataBuffer:
            # Couldn't send data
            result = 0
            break
        else:
            result = None
        self.offset += l

        if len(self.dataBuffer) - self.offset + self._tempDataLen < self.bufferSize:
          # quit, 'cause dataBuffer AND _tempDataBuffer is small enough
          break
    #
    # If there is nothing left to send,
    if self.offset == len(self.dataBuffer) and not self._tempDataLen:
        self.dataBuffer = """"
        self.offset = 0
        # stop writing.
        self.stopWriting()
        # If I've got a producer who is supposed to supply me with data,
        if self.producer is not None and ((not self.streamingProducer)
                                          or self.producerPaused):
            # tell them to supply some more.
            self.producerPaused = 0
            self.producer.resumeProducing()
        elif self.disconnecting:
            # But if I was previously asked to let the connection die, do
            # so.
            return self._postLoseConnection()
        elif self._writeDisconnecting:
            # I was previously asked to to half-close the connection.
            result = self._closeWriteConnection()
            self._writeDisconnected = True
            return result
    return result
}}}

thanks, GThomas",enhancement,new,low,,core,,"MemoryError, FileDescriptor",,,,