[Twisted-Python] Sending a long string/buffer without copying it
ellisonbg.net at gmail.com
Thu Sep 28 18:48:08 EDT 2006
But a producer will just make sure the whole thing isn't copied at the
same time right? It still does many smaller copies - while the memory
is saved there is still the performance hit.
I just wanted to make sure that I wan't missing something obvious. I
think the right way of doing this is to use a true rw buffer, such as
those created by numpy.
On 9/28/06, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> On Thu, 28 Sep 2006 00:11:41 -0600, Brian Granger <ellisonbg.net at gmail.com> wrote:
> >In one of my Twisted based applications, I need to send large string
> >and buffers. They can be 100's of MB's long (they come from large
> >numpy arrays). I would like to be able to send them *without making
> >any copies* in the process.
> >This seems to be dificult with the way that certain parts of Twisted
> >are written:
> >in protocols.basic many of the sendString/sendLine method having
> >things that make a copy of the string or line to be send:
> > def sendLine(self, line):
> > """Sends a line to the other end of the connection.
> > """
> > return self.transport.write(line + self.delimiter)
> >If line is 100MB, this just made a second 100MB string. To make
> >things worse, in my case a server needs to send this line to many
> >clients that are connected. The line gets copied for each client! If
> >I have 10 clients, I have nearly a GB worth of extra memory allocated
> >for this temporary copy.
> >This problem is easy solve at the protocol level: you just do separate
> >writes for the delimiter and the line. Or if you are using a length
> >prefixed protocol, write the length bytes and the string separately.
> >Even if I do that, it appears that Twisted is making copies elsewhere
> >- like in FileDescriptor.doWrite. So, how can I send something
> >without making a copy? I don't mind making copies of slices, just not
> >the whole thing.
> Don't pass the entire thing to a single call to transport.write() (or
> LineReceiver.sendLine). Instead, write a producer.
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
More information about the Twisted-Python