[Twisted-Python] producer / consumer question

[glyph at divmod.com]

On 04:13 pm, markus at bluegap.ch wrote:
>I have yet another question...  a producer has to learn about it's 
>consumer somehow, otherwise it would not know where to write it's data 
>to.

>Why is that? What's the best way to associate a producer with a 
>consumer? Is something like the following okay:

This is a really deep problem, unfortunately, and Valentino has 
correctly identified that it's the same one I was going to talk about 
when responding to his twisted-web posting :).

Your solution doesn't really solve anything, since neither producers nor 
consumers are documented to have the attributes you are assigning, nor 
do anything with them.

Consumers are documented to have a "write" method, but the documentation 
is misleading.  The most common type of producer - tcp.Connection - does 
not put data into its consumer's "write" method, it calls its protocol's 
dataReceived method.  Many other producers are actually written 
specifically to call higher-level methods on something other than their 
consumer (such as callRemote) rather than "write".

The basic use-case here is that the responsibility for producing bytes 
and the responsibility for throttling data production are usually 
separated in an application.  I agree that this is not really the best 
way to separate them.  The API grew up almost by accident, it wasn't 
really designed intentionally, although it has worked surprisingly well 
considering.

In many cases it does seem that we want a high-level API for setting up 
pipelines of producer-to-consumer-to-producer, or (in the case of web2 
streams) end-to-end producer-producer-producer sequences.  While many of 
these use-cases are superficially similar, my own attempts to resolve 
this shows that there is a surprising diversity of requirements which 
are difficult to resolve simultaneously by "simply fixing" the existing 
consumer / producer mess.

Keep in mind, also, that a great deal of code has been written (and will 
be written before everyone has upgraded to the next version of Twisted) 
that uses the current IConsumer/IProducer APIs, and all of that will 
need to keep working for the forseeable future.

You can find some discussion, as well as our attempts to create 
something better, here:

    http://twistedmatrix.com/trac/ticket/1956

If you have new information, feel free to comment there, but please read 
all the comments and try to understand them first: there has been a 
*lot* of back-and-forth over this issue.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20070704/2c651125/attachment.html>