[Twisted-Python] COMMERCIAL:Re: Clarification or pausing a consumer

[Barry Scott]

On Monday, 23 October 2017 17:31:19 BST Jean-Paul Calderone wrote:
> On Mon, Oct 23, 2017 at 8:10 AM, Barry Scott <barry.scott at forcepoint.com>
> 
> wrote:
> > On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
> > > > On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone
> > > > <exarkun at twistedmatrix.com> wrote:
> > > > 
> > > > On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <
> > 
> > barry.scott at forcepoint.com
> > 
> > > > <mailto:barry.scott at forcepoint.com>> wrote: I have a app that needs to
> > > > send data as an http response. But the data is produced slowly. More
> > > > slowly then the time taken to transmit it.
> > > > 
> > > > The consumer keeps calling resumeProducing, but there is no data to
> > 
> > send
> > 
> > > > often.
> > > > 
> > > > This sounds like a bug in the consumer.  There are two cases.
> > > > 
> > > > If the producer is "streaming" or "push" (two words, one meaning) then
> > 
> > the
> > 
> > > > consumer should call resumeProducing once and let data come.  If it
> > 
> > wants
> > 
> > > > to slow down the data, it can call pauseProducing.  There's nothing
> > 
> > else
> > 
> > > > it should be doing.
> > > 
> > > This is the good kind of producer.
> > > 
> > > > If the producer is "not streaming" or "pull" (again, one meaning) then
> > 
> > the
> > 
> > > > consumer should call resumeProducing and wait for a write call.  Then
> > 
> > it
> > 
> > > > should call resumeProducing again (if it wants more data).  It should
> > 
> > not
> > 
> > > > call resumeProducing again while waiting for a write call.
> > > 
> > > This is the bad kind.  It is basically pointless.  It was a design error
> > 
> > to
> > 
> > > include it and it should be removed from Twisted eventually.
> > 
> > Oh. This is surprising to here what is the rational behind this "good" vs.
> > "bad"?
> 
> What makes it "bad" is that it creates extra work for every consumer
> implementation.  The extra work is implementing largely the same thing
> every time so it's also redundant work.  The logic for knowing when to call
> resumeProducing again is always exactly the same regardless of the
> consumer.  It doesn't belong inside every consumer.

o.k. I see. So you want to have the consumer logic implemented once and
reused as needed?

> 
> > My use case is that I'm a proxy and I can only produce what has been
> > received
> > from the other side.
> 
> This sounds like it should be a push producer to me.

Yes it is.

> 
> > A hard CPU loop calling resumeProducing is not appropiate for this use
> > case.
> 
> This is appropriate behavior for neither kind of producer.  As I wrote in
> my first reply, a consumer with a pull ("not streaming", "bad") producer
> should call resumeProducing once and then wait until write is called on
> it.  After write is called on it, it may call resumeProducing again if it
> wants.

But that is what happens is you convert a push to a pull which is why I 
noticed this.

>> Related to that why does HTTPChannel.registerProducer convert
>> a IPullProducer into a IPushProducer using _PullToPush?

It sure looks like a bug to give me that API that is the shape to take push or 
pull but inside then force into pull.

Barry

> 
> Jean-Paul
> 
> > I had to work around what looked like a bug that the streaming parameter
> > is
> > ignored and call unregisterProducer/registedProduced to get back usable
> > behaviour.
> > 
> > Barry
> > 
> > > > What is the correct way to pause the consumer so that it does not keep
> > > > calling resumeProducing? unregisterProducer?
> > > > 
> > > > Probably just fix the consumer implementation to not be broken.  Or
> > 
> > switch
> > 
> > > > to tubes which has a simpler model and (in principle) fewer places to
> > > > make mistakes.
> > > 
> > > My guess is not that the consumer is broken, but rather, you registered
> > 
> > your
> > 
> > > producer as the bad kind of producer ("pull") and it's just doing as it
> > 
> > was
> > 
> > > told, which is to keep calling resumeProducing over and over again every
> > > time it wants data.
> > > 
> > > Let us know how it works out!
> > > 
> > > > Then when the next block of data is available what is the correct way
> > 
> > to
> > 
> > > > resume the consumer? registerProducer?
> > > > 
> > > > Related to that why does HTTPChannel.registerProducer convert a
> > > > IPullProducer into a IPushProducer using _PullToPush?
> > > > 
> > > > Probably as an attempt to simplify the implementation - to make it so
> > 
> > the
> > 
> > > > consumer can pretend there's just one kind of producer instead of two
> > > > (which would be nice for everyone).
> > > > 
> > > > Jean-Paul
> > > > 
> > > > _______________________________________________
> > > > Twisted-Python mailing list
> > > > Twisted-Python at twistedmatrix.com
> > > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> > 
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
> Scanned by Forcepoint Email Security Gateway
> 
> To report this email as SPAM, please forward it to spam at forcepoint.com