[Twisted-Python] COMMERCIAL:Re: Clarification or pausing a consumer

[Barry Scott]

On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
> > On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone
> > <exarkun at twistedmatrix.com> wrote:
> > 
> > On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <barry.scott at forcepoint.com
> > <mailto:barry.scott at forcepoint.com>> wrote: I have a app that needs to
> > send data as an http response. But the data is produced slowly. More
> > slowly then the time taken to transmit it.
> > 
> > The consumer keeps calling resumeProducing, but there is no data to send
> > often.
> > 
> > This sounds like a bug in the consumer.  There are two cases.
> > 
> > If the producer is "streaming" or "push" (two words, one meaning) then the
> > consumer should call resumeProducing once and let data come.  If it wants
> > to slow down the data, it can call pauseProducing.  There's nothing else
> > it should be doing.
> This is the good kind of producer.
> 
> > If the producer is "not streaming" or "pull" (again, one meaning) then the
> > consumer should call resumeProducing and wait for a write call.  Then it
> > should call resumeProducing again (if it wants more data).  It should not
> > call resumeProducing again while waiting for a write call.
> This is the bad kind.  It is basically pointless.  It was a design error to
> include it and it should be removed from Twisted eventually.

Oh. This is surprising to here what is the rational behind this "good" vs. 
"bad"?

My use case is that I'm a proxy and I can only produce what has been received 
from the other side.

A hard CPU loop calling resumeProducing is not appropiate for this use case.

I had to work around what looked like a bug that the streaming parameter is 
ignored and call unregisterProducer/registedProduced to get back usable 
behaviour.

Barry

> > What is the correct way to pause the consumer so that it does not keep
> > calling resumeProducing? unregisterProducer?
> > 
> > Probably just fix the consumer implementation to not be broken.  Or switch
> > to tubes which has a simpler model and (in principle) fewer places to
> > make mistakes.
> My guess is not that the consumer is broken, but rather, you registered your
> producer as the bad kind of producer ("pull") and it's just doing as it was
> told, which is to keep calling resumeProducing over and over again every
> time it wants data.
> 
> Let us know how it works out!
> 
> > Then when the next block of data is available what is the correct way to
> > resume the consumer? registerProducer?
> > 
> > Related to that why does HTTPChannel.registerProducer convert a
> > IPullProducer into a IPushProducer using _PullToPush?
> > 
> > Probably as an attempt to simplify the implementation - to make it so the
> > consumer can pretend there's just one kind of producer instead of two
> > (which would be nice for everyone).
> > 
> > Jean-Paul
> > 
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python