Ticket #3682 enhancement new
twisted.internet.abstract.FileDescriptor accumulates _tempData and throws MemoryError
|Reported by:||gthomas||Owned by:||gthomas|
twisted.internet.abstract.FileDescriptor.doWrite method sends some data, but doesn't care about _tempDataBuffer's size.
In the write method, it tries to pause the producer, but too much ifs are there (self.provider exists and it is a not a StreamProducer)).
So when the producer produces too much data, the
"".join(self._tempDataBuffer) #line 101
throws a MemoryError.
- if self.dataBuffer is small, only just a few chunks should be appended to it from self._tempDataBuffer
- the doWrite method should try to slim self._tempDataBuffer at least till self.bufferSize (so too much data just can't accumulate in self.dataBuffer and self._tempDataBuffer).
A possible solution (twisted.internet.abstract.FileDescriptor.doWrite:
def doWrite(self): """Called when data can be written. A result that is true (which will be a negative number) implies the connection was lost. A false result implies the connection is still there; a result of 0 implies no write was done, and a result of None indicates that a write was done. """ #LOG.debug('%s.doWrite: db=%d tdb=%d' # % (self, len(self.dataBuffer)-self.offset, self._tempDataLen)) while 1: # always try to do something # quit when dataBuffer size is acceptably low (see end) if len(self.dataBuffer) - self.offset < self.SEND_LIMIT: # If there is currently less than SEND_LIMIT bytes left to send # in the string, extend it with the array data. buf =  accum = len(self.dataBuffer)-self.offset m = 2 * max(self.SEND_LIMIT, self.bufferSize) chunk = None while self._tempDataBuffer and accum < m: # don't want to join too much data at once chunk = self._tempDataBuffer.pop(0) buf.append(chunk) n = len(chunk) accum += n self._tempDataLen -= n #LOG.debug('%s.doWrite: accum=%d tdb=%d' % (self, accum, self._tempDataLen,)) del chunk, accum, m self.dataBuffer = (buffer(self.dataBuffer, self.offset) + "".join(buf)) del buf self.offset = 0 # Send as much data as you can. if self.offset: l = self.writeSomeData(buffer(self.dataBuffer, self.offset)) else: l = self.writeSomeData(self.dataBuffer) # There is no writeSomeData implementation in Twisted which returns # 0, but the documentation for writeSomeData used to claim negative # integers meant connection lost. Keep supporting this here, # although it may be worth deprecating and removing at some point. if l < 0 or isinstance(l, Exception): return l if l == 0 and self.dataBuffer: # Couldn't send data result = 0 break else: result = None self.offset += l if len(self.dataBuffer) - self.offset + self._tempDataLen < self.bufferSize: # quit, 'cause dataBuffer AND _tempDataBuffer is small enough break # # If there is nothing left to send, if self.offset == len(self.dataBuffer) and not self._tempDataLen: self.dataBuffer = "" self.offset = 0 # stop writing. self.stopWriting() # If I've got a producer who is supposed to supply me with data, if self.producer is not None and ((not self.streamingProducer) or self.producerPaused): # tell them to supply some more. self.producerPaused = 0 self.producer.resumeProducing() elif self.disconnecting: # But if I was previously asked to let the connection die, do # so. return self._postLoseConnection() elif self._writeDisconnecting: # I was previously asked to to half-close the connection. result = self._closeWriteConnection() self._writeDisconnected = True return result return result