[Twisted-Python] Producers and Consumers

Andrew Bennetts andrew-twisted at puzzling.org
Wed Jan 15 07:08:13 MST 2003


Hi all,

It seems to me that the IProducer and IConsumer interfaces in
twisted.internet.interfaces are effectively useless.

Say you wanted to write a class with a method like:

class MyConsumer:
    def foo(self, producer):
        """@type producer: L{IProducer}"""
        # keep a reference to the producer, to consume stuff from it later

How do you actually implement that method?

IProducer only specifies {resume,pause,stop}Producing.  IConsumer states
that it expects its write() method to be called by the producer.  But there
is no way to find out the IConsumer's producer using the methods documented
in IProducer/IConsumer (the interface doesn't explicitly say it, but it's
fairly obvious that there can only be one consumer per producer, and vice
versa).

Similarly, you can't write a method like this:

    def streamData(self):
        """@returns: a L{Deferred}, which will be called with an object
                     implementing L{IProducer}."""
        # ...

You'd use this like:

    d = source.streamData()
    d.addCallback(lambda p: self.registerProducer(p, streaming=1))

But of course, without knowing something extra about the producer beyond
what IProducer tells you, you can't implement this, because there's no way
to tell an IProducer what IConsumer to write to.

(Incidentally, the 'streaming' argument to registerProducer isn't very
clearly explained... after several readings, I *think* all it means is that
the producer starts producing immediately, rather than waiting for a call to
resumeProducing.)

Finally, I also see that there is no way for an producer to signal to its
consumer that it has finished.  Perhaps for the intended use cases of
producers and consumers, this is superfluous, but it would be very useful to
me!

My motivation for all this (in case you hadn't guessed :), is FTP.  I've
previously discussed the way the code currently works (by taking a Protocol
instance as an argument, so that downloads can be fed directly to it), and
I'm looking for clean ways to improve it, so I can add the STOR command
(i.e. uploads).

Now, I could do FTPClient.store similarly to how I do FTPClient.retrieve,
i.e.

    def store(self, path, protocol):
        """ ...
        @param path: The path to store data to.
        @param protocol: A L{Protocol} instance that will write the data
            to be stored upon a call to C{connectionMade}, and will call
            C{self.transport.loseConnection()} when done.
        """
        # ...

But everyone assures me that requiring protocol instances as parameters is a
bad way to do things, and my gut feeling agrees -- I just don't feel
comfortable with asking Twisted users to instantiate a Protocol without
involving a Factory.

The obvious alternative that occured to me was to use producers and
consumers -- after all, a download is just a stream of bytes waiting to be
consumed, and an upload is just a connection waiting for data to be produced
-- but IConsumer and IProducer don't fit this use-case.

So, are IConsumer and IProducer fundamentally broken, or is it intentional
that all Consumer/Producer implementations are co-dependent?  And what is
the right interface for FTPClient?

If I'm missing the Zen of Consumption and Production, then please enlighten
me, so that I may improve the existing docstrings and maybe write a howto?
:)

Regards,

-Andrew.





More information about the Twisted-Python mailing list