[Twisted-Python] COMMERCIAL:Re: Clarification or pausing a consumer

[Jean-Paul Calderone]

On Mon, Oct 23, 2017 at 8:10 AM, Barry Scott <barry.scott at forcepoint.com>
wrote:

> On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
> > > On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone
> > > <exarkun at twistedmatrix.com> wrote:
> > >
> > > On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <
> barry.scott at forcepoint.com
> > > <mailto:barry.scott at forcepoint.com>> wrote: I have a app that needs to
> > > send data as an http response. But the data is produced slowly. More
> > > slowly then the time taken to transmit it.
> > >
> > > The consumer keeps calling resumeProducing, but there is no data to
> send
> > > often.
> > >
> > > This sounds like a bug in the consumer.  There are two cases.
> > >
> > > If the producer is "streaming" or "push" (two words, one meaning) then
> the
> > > consumer should call resumeProducing once and let data come.  If it
> wants
> > > to slow down the data, it can call pauseProducing.  There's nothing
> else
> > > it should be doing.
> > This is the good kind of producer.
> >
> > > If the producer is "not streaming" or "pull" (again, one meaning) then
> the
> > > consumer should call resumeProducing and wait for a write call.  Then
> it
> > > should call resumeProducing again (if it wants more data).  It should
> not
> > > call resumeProducing again while waiting for a write call.
> > This is the bad kind.  It is basically pointless.  It was a design error
> to
> > include it and it should be removed from Twisted eventually.
>
> Oh. This is surprising to here what is the rational behind this "good" vs.
> "bad"?
>

What makes it "bad" is that it creates extra work for every consumer
implementation.  The extra work is implementing largely the same thing
every time so it's also redundant work.  The logic for knowing when to call
resumeProducing again is always exactly the same regardless of the
consumer.  It doesn't belong inside every consumer.


>
> My use case is that I'm a proxy and I can only produce what has been
> received
> from the other side.
>
>
This sounds like it should be a push producer to me.


> A hard CPU loop calling resumeProducing is not appropiate for this use
> case.
>
>
This is appropriate behavior for neither kind of producer.  As I wrote in
my first reply, a consumer with a pull ("not streaming", "bad") producer
should call resumeProducing once and then wait until write is called on
it.  After write is called on it, it may call resumeProducing again if it
wants.

Jean-Paul


> I had to work around what looked like a bug that the streaming parameter is
> ignored and call unregisterProducer/registedProduced to get back usable
> behaviour.
>
> Barry
>
> > > What is the correct way to pause the consumer so that it does not keep
> > > calling resumeProducing? unregisterProducer?
> > >
> > > Probably just fix the consumer implementation to not be broken.  Or
> switch
> > > to tubes which has a simpler model and (in principle) fewer places to
> > > make mistakes.
> > My guess is not that the consumer is broken, but rather, you registered
> your
> > producer as the bad kind of producer ("pull") and it's just doing as it
> was
> > told, which is to keep calling resumeProducing over and over again every
> > time it wants data.
> >
> > Let us know how it works out!
> >
> > > Then when the next block of data is available what is the correct way
> to
> > > resume the consumer? registerProducer?
> > >
> > > Related to that why does HTTPChannel.registerProducer convert a
> > > IPullProducer into a IPushProducer using _PullToPush?
> > >
> > > Probably as an attempt to simplify the implementation - to make it so
> the
> > > consumer can pretend there's just one kind of producer instead of two
> > > (which would be nice for everyone).
> > >
> > > Jean-Paul
> > >
> > > _______________________________________________
> > > Twisted-Python mailing list
> > > Twisted-Python at twistedmatrix.com
> > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20171023/5b31fecd/attachment-0002.html>