[Twisted-Python] How do you determine the buffer size of a transport - a use-case for not using back pressure

[Glyph Lefkowitz]

Hi Steve,

It looks like I had marked this message as interesting and warranting a reply, but never got around to it.  I'm sorry it's been quite a while!  I appreciate the amount of research you did here :-).

> On Aug 17, 2016, at 3:43 PM, Steve Morin <steve.morin at gmail.com> wrote:
> 
> Twisted Community
> 
> Problem: How do you determine the buffer size of a transport, to know how much data is waiting to be transmitted from using transport.write?
> 
> Wait! You're going to say: use the Producer Consumer API ( http://twistedmatrix.com/documents/current/core/howto/producers.html <http://twistedmatrix.com/documents/current/core/howto/producers.html> )

This is, unfortunately, the only solution :).

> To do what: So that instead of using back pressure I can check the buffer and when it's "too big/full" can decide to do something to the transport I am writing to:

I think when you say "back pressure" you're referring to your program exerting back-pressure on its peer.  I understand why you don't want to do that.  However, there's another kind of back pressure - your peer exerting back pressure on your program.

Commensurately, there are two ways to use back pressure:

To exert back pressure on your peer, call `self.transport.pauseProducing()`.  Later, when you're ready to receive more data, call `self.transport.resumeProducing()`.  This is what you don't want to do.
To detect when back pressure is applied from your peer, call `self.transport.registerProducer(self, True)`; then the reactor will call pauseProducing() when its buffer is full and and resumeProducing() when it empties out again.

Your list of things you might want to do here:

> - Buffer to disk instead of memory
> - Kill the transport
> - Decide to skip sending some data
> - Send an error or message to the transport I am writing to
> - Reduce the resolution, increase the compression (things like video or audio)

is a good one, and all these things can be achieved.  Going through them:

If you want to buffer to disk instead of memory, have a method like:

def someDataToSendToPeer(self, someData):
    if self.isProducing:
        self.transport.write(someData)
    else:
        self.bufferFile.write(someData)

def pauseProducing(self):
    self.isProducing = False
    self.bufferFile = open("buffer.file", "wb")

def resumeProducing(self):
    self.isProducing = True
    self.startUbufferingFromFile()

If you want to kill the transport,

def pauseProducing(self):
    self.transport.abortConnection()

If you want to reduce video stream quality,

def streamSomeRawVideo(self, someRawVideo):
    if self.isProducing:
        self.transport.write(self.videoBuffer.addAndEncodeToBytes(someRawVideo))
    else:
        self.videoBuffer.addAndCompressSomeMore(someRawVideo)

and so on, and so on.

Basically, you can treat the buffer as "empty" until pauseProducing() is called.  Once it is, you can treat it as "full".

Hope this was helpful, and still timely enough for you to make some use of it :).

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20160928/99042030/attachment-0001.html>