[Twisted-Python] Sending a long string/buffer without copying it
ellisonbg.net at gmail.com
Thu Sep 28 02:11:41 EDT 2006
In one of my Twisted based applications, I need to send large string
and buffers. They can be 100's of MB's long (they come from large
numpy arrays). I would like to be able to send them *without making
any copies* in the process.
This seems to be dificult with the way that certain parts of Twisted
in protocols.basic many of the sendString/sendLine method having
things that make a copy of the string or line to be send:
def sendLine(self, line):
"""Sends a line to the other end of the connection.
return self.transport.write(line + self.delimiter)
If line is 100MB, this just made a second 100MB string. To make
things worse, in my case a server needs to send this line to many
clients that are connected. The line gets copied for each client! If
I have 10 clients, I have nearly a GB worth of extra memory allocated
for this temporary copy.
This problem is easy solve at the protocol level: you just do separate
writes for the delimiter and the line. Or if you are using a length
prefixed protocol, write the length bytes and the string separately.
Even if I do that, it appears that Twisted is making copies elsewhere
- like in FileDescriptor.doWrite. So, how can I send something
without making a copy? I don't mind making copies of slices, just not
the whole thing.
More information about the Twisted-Python