[Twisted-Python] Synchronous byte streaming

[exarkun at twistedmatrix.com]

On 03:10 pm, darren at ontrenet.com wrote:
>Hi,
>  I received some great advice in the past about using
>Int32StringReceiver to be able to send a reliable message of bytes
>from the client to the server without hassling with unordered or 
>partial
>messages. Seems to work.
>
>But I have one problem. My client iterates over a file sending 7MB byte
>messages to transport.sendString() to a server.
>What i see is that the client seems to buffer or wait to send those.
>After a few hundred sendStrings() still the server has not received one
>message. The result is my client gets throttled bad in terms of disk 
>and
>memory. After all my sendString writes, then eventually the server
>receives the first message. But my computer is 100% max'd out.  highly
>undesirable.
>
>What I want is for each on-the-wire send to complete, before sendString
>returns so resources can be managed properly.
>
>Is there _any_ way in Twisted to do this?

You want to make your client use the producer/consumer APIs.  You can 
find some docs here:

  http://twistedmatrix.com/documents/current/core/howto/producers.html

Here's a simple example of what you might do:

    class SendSomeStuff(Int32StringReceiver):
        def connectionMade(self):
            self.transport.registerProducer(self, False)

        def resumeProducing(self):
            self.sendString(self.someFileObj.read(2 ** 16))


This is missing error handling some of the other required parts of the 
producer interface, but it should get you started.

Jean-Paul