[Twisted-Python] producer / consumer question
glyph at divmod.com
glyph at divmod.com
Wed Jul 4 15:01:51 EDT 2007
On 04:13 pm, markus at bluegap.ch wrote:
>I have yet another question... a producer has to learn about it's
>consumer somehow, otherwise it would not know where to write it's data
>Why is that? What's the best way to associate a producer with a
>consumer? Is something like the following okay:
This is a really deep problem, unfortunately, and Valentino has
correctly identified that it's the same one I was going to talk about
when responding to his twisted-web posting :).
Your solution doesn't really solve anything, since neither producers nor
consumers are documented to have the attributes you are assigning, nor
do anything with them.
Consumers are documented to have a "write" method, but the documentation
is misleading. The most common type of producer - tcp.Connection - does
not put data into its consumer's "write" method, it calls its protocol's
dataReceived method. Many other producers are actually written
specifically to call higher-level methods on something other than their
consumer (such as callRemote) rather than "write".
The basic use-case here is that the responsibility for producing bytes
and the responsibility for throttling data production are usually
separated in an application. I agree that this is not really the best
way to separate them. The API grew up almost by accident, it wasn't
really designed intentionally, although it has worked surprisingly well
In many cases it does seem that we want a high-level API for setting up
pipelines of producer-to-consumer-to-producer, or (in the case of web2
streams) end-to-end producer-producer-producer sequences. While many of
these use-cases are superficially similar, my own attempts to resolve
this shows that there is a surprising diversity of requirements which
are difficult to resolve simultaneously by "simply fixing" the existing
consumer / producer mess.
Keep in mind, also, that a great deal of code has been written (and will
be written before everyone has upgraded to the next version of Twisted)
that uses the current IConsumer/IProducer APIs, and all of that will
need to keep working for the forseeable future.
You can find some discussion, as well as our attempts to create
something better, here:
If you have new information, feel free to comment there, but please read
all the comments and try to understand them first: there has been a
*lot* of back-and-forth over this issue.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python