[Twisted-Python] Sending a long string/buffer without copying it
exarkun at divmod.com
Thu Sep 28 08:22:01 EDT 2006
On Thu, 28 Sep 2006 00:11:41 -0600, Brian Granger <ellisonbg.net at gmail.com> wrote:
>In one of my Twisted based applications, I need to send large string
>and buffers. They can be 100's of MB's long (they come from large
>numpy arrays). I would like to be able to send them *without making
>any copies* in the process.
>This seems to be dificult with the way that certain parts of Twisted
>in protocols.basic many of the sendString/sendLine method having
>things that make a copy of the string or line to be send:
> def sendLine(self, line):
> """Sends a line to the other end of the connection.
> return self.transport.write(line + self.delimiter)
>If line is 100MB, this just made a second 100MB string. To make
>things worse, in my case a server needs to send this line to many
>clients that are connected. The line gets copied for each client! If
>I have 10 clients, I have nearly a GB worth of extra memory allocated
>for this temporary copy.
>This problem is easy solve at the protocol level: you just do separate
>writes for the delimiter and the line. Or if you are using a length
>prefixed protocol, write the length bytes and the string separately.
>Even if I do that, it appears that Twisted is making copies elsewhere
>- like in FileDescriptor.doWrite. So, how can I send something
>without making a copy? I don't mind making copies of slices, just not
>the whole thing.
Don't pass the entire thing to a single call to transport.write() (or
LineReceiver.sendLine). Instead, write a producer.
More information about the Twisted-Python