[Twisted-Python] COMMERCIAL:Re: Clarification or pausing a consumer
Barry Scott
barry.scott at forcepoint.com
Tue Oct 24 04:10:30 MDT 2017
On Monday, 23 October 2017 17:31:19 BST Jean-Paul Calderone wrote:
> On Mon, Oct 23, 2017 at 8:10 AM, Barry Scott <barry.scott at forcepoint.com>
>
> wrote:
> > On Saturday, 14 October 2017 07:44:12 BST Glyph wrote:
> > > > On Oct 13, 2017, at 5:47 PM, Jean-Paul Calderone
> > > > <exarkun at twistedmatrix.com> wrote:
> > > >
> > > > On Fri, Oct 13, 2017 at 8:29 AM, Barry Scott <
> >
> > barry.scott at forcepoint.com
> >
> > > > <mailto:barry.scott at forcepoint.com>> wrote: I have a app that needs to
> > > > send data as an http response. But the data is produced slowly. More
> > > > slowly then the time taken to transmit it.
> > > >
> > > > The consumer keeps calling resumeProducing, but there is no data to
> >
> > send
> >
> > > > often.
> > > >
> > > > This sounds like a bug in the consumer. There are two cases.
> > > >
> > > > If the producer is "streaming" or "push" (two words, one meaning) then
> >
> > the
> >
> > > > consumer should call resumeProducing once and let data come. If it
> >
> > wants
> >
> > > > to slow down the data, it can call pauseProducing. There's nothing
> >
> > else
> >
> > > > it should be doing.
> > >
> > > This is the good kind of producer.
> > >
> > > > If the producer is "not streaming" or "pull" (again, one meaning) then
> >
> > the
> >
> > > > consumer should call resumeProducing and wait for a write call. Then
> >
> > it
> >
> > > > should call resumeProducing again (if it wants more data). It should
> >
> > not
> >
> > > > call resumeProducing again while waiting for a write call.
> > >
> > > This is the bad kind. It is basically pointless. It was a design error
> >
> > to
> >
> > > include it and it should be removed from Twisted eventually.
> >
> > Oh. This is surprising to here what is the rational behind this "good" vs.
> > "bad"?
>
> What makes it "bad" is that it creates extra work for every consumer
> implementation. The extra work is implementing largely the same thing
> every time so it's also redundant work. The logic for knowing when to call
> resumeProducing again is always exactly the same regardless of the
> consumer. It doesn't belong inside every consumer.
o.k. I see. So you want to have the consumer logic implemented once and
reused as needed?
>
> > My use case is that I'm a proxy and I can only produce what has been
> > received
> > from the other side.
>
> This sounds like it should be a push producer to me.
Yes it is.
>
> > A hard CPU loop calling resumeProducing is not appropiate for this use
> > case.
>
> This is appropriate behavior for neither kind of producer. As I wrote in
> my first reply, a consumer with a pull ("not streaming", "bad") producer
> should call resumeProducing once and then wait until write is called on
> it. After write is called on it, it may call resumeProducing again if it
> wants.
But that is what happens is you convert a push to a pull which is why I
noticed this.
>> Related to that why does HTTPChannel.registerProducer convert
>> a IPullProducer into a IPushProducer using _PullToPush?
It sure looks like a bug to give me that API that is the shape to take push or
pull but inside then force into pull.
Barry
>
> Jean-Paul
>
> > I had to work around what looked like a bug that the streaming parameter
> > is
> > ignored and call unregisterProducer/registedProduced to get back usable
> > behaviour.
> >
> > Barry
> >
> > > > What is the correct way to pause the consumer so that it does not keep
> > > > calling resumeProducing? unregisterProducer?
> > > >
> > > > Probably just fix the consumer implementation to not be broken. Or
> >
> > switch
> >
> > > > to tubes which has a simpler model and (in principle) fewer places to
> > > > make mistakes.
> > >
> > > My guess is not that the consumer is broken, but rather, you registered
> >
> > your
> >
> > > producer as the bad kind of producer ("pull") and it's just doing as it
> >
> > was
> >
> > > told, which is to keep calling resumeProducing over and over again every
> > > time it wants data.
> > >
> > > Let us know how it works out!
> > >
> > > > Then when the next block of data is available what is the correct way
> >
> > to
> >
> > > > resume the consumer? registerProducer?
> > > >
> > > > Related to that why does HTTPChannel.registerProducer convert a
> > > > IPullProducer into a IPushProducer using _PullToPush?
> > > >
> > > > Probably as an attempt to simplify the implementation - to make it so
> >
> > the
> >
> > > > consumer can pretend there's just one kind of producer instead of two
> > > > (which would be nice for everyone).
> > > >
> > > > Jean-Paul
> > > >
> > > > _______________________________________________
> > > > Twisted-Python mailing list
> > > > Twisted-Python at twistedmatrix.com
> > > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
> Scanned by Forcepoint Email Security Gateway
>
> To report this email as SPAM, please forward it to spam at forcepoint.com
More information about the Twisted-Python
mailing list