[Twisted-web] Web2 and Half-Baked Reinvention of Wheels

Valentino Volonghi aka Dialtone dialtone at divmod.com
Tue Jul 3 19:47:37 EDT 2007

On Tue, 03 Jul 2007 19:52:06 -0000, glyph at divmod.com wrote:

>If anyone out there shares his frustration and would like to help us along 
>towards a supported web2 release, or any other part of Twisted for that 
>matter, just have a look at the Twisted tracker and steal a ticket from some 
>of the obviously overcommitted and overworked maintainers. The milestone 
>link I posted is the best start for web2.

I'm already fixing some bugs with the current code as part of the SoC,
and I've currently fixed 4 or 5 of them (the last one waiting for
review in branch 1895). I'm currently starting, or I'm hoping to
start, work on no-stream-1937 branch...

>Many of those tickets are underspecified, but don't let that stop you. If 
>you find a ticket that you think sounds interesting but you don't understand 
>how to proceed on it, feel free to comment and ask for clarification - it is 
>often far easier for a maintainer to respond to a prompt for some answers to 
>questions than to drive tests, documentation, and code all the way through 
>the review process.

Since you kindly offered for clarifications:

I see one problem with the current IConsumer/IProducer API and it's
returning deferreds when a producer finished its job.

The current IConsumer API uses registerProducer() but doesn't allow to
return anything from the method, and in fact this is what the
transport does. Unfortunately it would be useful to deal with the
finish event of producing to a consumer for many reasons, like you
don't know in advance when the produced content will finish.

In Twisted this problem was solved, partially, by FileSender which uses
a beginFileTransfer(fobj, consumer) method to return a deferred that
fires at the end of the file object. Unfortunately this is not really
a reliable method since it's not part of the IProducer interface and
any producer might use whatever method they like and, worst of all,
I'd anyway need to rewrite or add a method to FileSender because I can't
really use beginFileTransfer() for a MemoryBufferProducer() object.

One of the needs is the chance to chain Producers/Consumers to do
input filtering and output filtering. While the first could also be
avoided (meaning that the current implementation is good enough IMHO),
output filtering is actually pretty cool to have since it allows the
user to dynamically add behavior to a request's response like it's
currently done.

Taking from JP's consumer-sketch.py code:

def _chain(hook, chainedElements):
    assert len(chainedElements) >= 2
    elements = iter(chainedElements)
    producer = elements.next()

    if isinstance(producer, Deferred):
        producer = yield producer

    for ele in elements:
        if isinstance(ele, Deferred):
            ele = yield ele
        finishedProducing = hook(producer, ele)
        producer = ele
    yield finishedProducing
_chain = defgen(_chain)

def forwardHook(producer, consumer):
    return producer.produceTo(consumer)

def chainOneWay(*x):

    Connect a producer to a consumer/producer to a consumer/producer to a ... ETC

    For example::

        a = FileSender('filename')
        b = GzipCompressor()
        c = ChunkEncoder()
        d = TCPTransport()

        finished = chainOneWay(a, b, c, d)
        def cbSentChunkedGzippedFile(ign):
            print 'Yaaaaaaaaaaaaay'

    Will do the obvious thing.
    return _chain(forwardHook, x)

This would be a really cool way to solve the problem and would also
present a pretty clean and simple API.

My question then is: how should I proceed? Clearly using this new API
is hard because everything else uses the current one, but the current
one lacks a fairly important functionality. Providing adapter in web2
for current producer/consumer stuff might seem overkill and not very
clean but might be a solution. Introducing this new API in Twisted
would be a nightmare. Is there any way to implement this using the
current IProducer/IConsumer API? I seek advice and some code examples
if anyone is willing to spend his mind on the problem a bit.

More information about the Twisted-web mailing list