Opened 13 years ago

Last modified 5 years ago

#1956 enhancement new

— at Make a less sucky producer/consumer APIVersion 16

Reported by: jknight Owned by: Glyph
Priority: normal Milestone:
Component: core Keywords:
Cc: Glyph, Jean-Paul Calderone, itamarst, radix, jknight, oubiwann, ashfall, ralphm, Tobias Oberstein, Julian Berman, free, daf Branch:
Author:

Description (last modified by Glyph)

The current producer/consumer API has a couple of problems:

  • It's impossible to get notifications of buffer size changes, which is important for certain kinds of timeout logic. For example, if you buffer a bunch of output, and then time out if the peer is inactive (isn't reading any data), then you need to know not just when the buffer is empty but when some data is consumed.
  • The streaming flag doesn't really make sense. It complicates the implementation of nested producers. While it doesn't actually break anything, it's just a bug. There's no reason to have it and it should be removed.
  • It's awkward to chain producers and consumers with each other (XXX: this problem needs better documentation, use case, example of awkward code).
  • IProtocol, ITransport, IProducer and IConsumer all have redundant "here's some data" / "there will be no more data" methods which are subtly different. These should be unified so that the protocol consuming data from the transport and the transport consuming data from the protocol look the same.

We should design an API which does not have these problems.

Change History (16)

comment:1 Changed 13 years ago by jknight

Cc: itamarst added; Itamar Turner-Trauring removed

comment:2 Changed 13 years ago by jknight

Here's a rough sketch to get started with.

Producer:

  • beProducing() -> Deferred: register the consumer and return a d called back when done producing (or errored), but don't start writing (badly named function)
  • bufferFull() -> None: when called, _MUST_ not call writeFn again. Called on state change only
  • bufferEmpty(writeFn) -> None: you can start writing data by calling writeFn. Called on state change only.
  • abort(failure): close down any internal state, and errback the deferred with the given failure.
  • split(offset) -> IProducer, IProducer: return two producers, one that stops at offset bytes, one that starts there. (optional? provided by ABC?). Only callable while not producing?

Consumer:

  • consumeFrom(Producer) -> Deferred: registers producer, adds internal callback to prod's returned deferred, and returns the d from the producer.

comment:3 Changed 13 years ago by jknight

Here's a rough sketch to get started with, but readable.

Producer:

  • beProducing() -> Deferred: register the consumer and return a d called back when done producing (or errored), but don't start writing (badly named function)
  • bufferFull() -> None: when called, _MUST_ not call writeFn again. Called on state change only
  • bufferEmpty(writeFn) -> None: you can start writing data by calling writeFn. Called on state change only.
  • abort(failure): close down any internal state, and errback the deferred with the given failure.
  • split(offset) -> IProducer, IProducer: return two producers, one that stops at offset bytes, one that starts there. (optional? provided by ABC?). Only callable while not producing?

Consumer:

  • consumeFrom(Producer) -> Deferred: registers producer, adds internal callback to prod's returned deferred, and returns the d from the producer.

comment:4 Changed 13 years ago by Glyph

Hooray a ticket.

AMP's streaming stuff will need something like this too. I will make a separate ticket for that with a detailed description tonight (I will try to do it with the current cons/prod API first).

I assume beProducing() should actually be produceTo(consumer) or something like that?

Can we modify this so that there are no Deferreds in the lowest level of the API? Instead, add finishedConsuming(consumer) and finishedProducing(producer) and APIs to Producer and Consumer respectively?

Sometimes, the producer wants to be "on top" (you have some data, you want to write it somewhere). Sometimes, the consumer wants to be "on top" (you want to get some data and you want to pass along a place to put it). How do we reconcile those two use cases? It sounds like it's pretty close in this sketch, but some detailed description of how beProducing and consumeFrom interact would be good.

comment:5 Changed 13 years ago by Glyph

To clarify, I'm not saying there should be no Deferreds anywhere, just that they shouldn't be the "primitive" here. Convenience APIs that return Deferreds are good, and absolutely necessary for short scripts.

comment:6 Changed 13 years ago by Glyph

Priority: normalhighest

comment:7 Changed 13 years ago by Glyph

Cc: radix added

comment:8 Changed 13 years ago by jknight

I was wondering about how you were going to do AMP's streaming, because it's almost exactly the same issues as I ran into for web2's streams..

I'm pretty sure calling it produceTo is not a good idea, because it doesn't actually take a consumer as an argument. One thing I like about my proposal is that the consumer actually has no API from the producer's standpoint. I think that is a good feature and shouldn't be discarded without thinking hard about it.

"beProducing" should never be called directly, it should always be called through a consumer's consumeFrom function. If you're passing around a data stream (producer), you call transport.consumeFrom(myDatastreamObject) when you finally find the place to put it. If you're passing around a data sink (consumer), you still call sink.consumeFrom(myDatastreamObject) when you finally find some data to send it. I don't think that's actually a problem?

It should've said:

Consumer:

  • consumeFrom(Producer) -> Deferred: registers producer, calls producer.beProducing(), adds internal callbacks to returned deferred (to deregister producer), and returns the d.

comment:9 Changed 13 years ago by jknight

To expand a bit on that, your finishedProducing(producer) would be the equivalent of calling back the deferred from beProducing on the producer. finishedConsuming() doesn't have any analog in the above API, and I can't see a need to have it, either. Either the consumer gets all the data streamed to it, or it pauses with bufferFull(), or it has an error and calls abort(). I'm not sure what a finishedConsuming() would be useful for?

comment:10 Changed 13 years ago by Glyph

Hmm. I hadn't considered the "Has no API" feature. That's pretty cool.

.split() bugs me though. I think that split() is actually an API that wants to be on the transport, .switchProtocol(newProto, pushbackData). I am kind of in a hurry right now but if that doesn't make sense I will expound later.

What I think I'm going to be doing for AMP's streaming is creating an argument type of "protocol factory", which takes a client protocol factory on the requesting side. If you want to upload, you pass that and then start writing when you get a successful response, if you want to download, you pass it and call connectionMade as soon as you get a success response from the responder, then wait for data.

comment:11 Changed 13 years ago by radix

Cc: jknight added

How about a P/C party day at Belmont and Highland this weekend? James, Glyph, Jp, Itamar?

comment:12 Changed 13 years ago by itamarst

saturday?

comment:13 Changed 13 years ago by radix

sounds good to me

comment:14 Changed 13 years ago by Jean-Paul Calderone

There's [source:/sandbox/exarkun/consumer-sketch.py some code] in svn now

comment:15 Changed 13 years ago by Glyph

OK so this weekend obviously isn't good but can we try to have another producer/consumer party on the 12th?

comment:16 Changed 13 years ago by Glyph

Description: modified (diff)

Updating the description a little bit (everybody else please feel free to add more).

Note: See TracTickets for help on using tickets.