[Twisted-web] streaming request (was: status of Twisted Web and Web 2)

Andrew McNabb amcnabb at mcnabbs.org
Thu Mar 6 09:15:53 EST 2008


On Wed, Mar 05, 2008 at 10:30:29PM -0500, Jean-Paul Calderone wrote:
>
> To clarify, this is about calling into application code in a web server
> once each time some new bytes from a request body are received from the
> client, right?

Exactly.

> Deferred.callback explicitly checks to see if you're giving it a Deferred
> and raises the TypeError.  The idea here is that most of the time, you're
> probably doing this by accident, not intentionally.  Clearly it wasn't an
> accident in your case.

I definitely was not doing it by accident. :)  Is this really a common
mistake?

> The way to actually get this behavior is to chain the Deferreds
> together.  Instead of a.callback(b), which I assume you was to send
> the result of b to the callbacks of a when they became available, you
> a.chainDeferred(b), which is really just a way to say
>
>  a.addCallbacks(b.callback, b.errback)
>
> However, I don't think this is generally the approach to take.

This actually isn't equivalent to what I was trying to do.  Think of my
approach this way.  As a user, you call:

d = streaming_download(url)

def got_some_data(value):
    new_data, d = value
    # add callback for the next time data get sent:
    d.addCallback(got_some_data)
    # do stuff with new_data

d.addCallback(got_some_data)


>> My second attempt was to create a new deferred each time data arrived
>> and to copy the callback list from the previous deferred.  This worked,
>> but it felt dirty, so I got rid of it.

Actually, after looking at my code again, it turns out that what I am
doing is actually a mix of attempt 2 and attempt 3.  This is the code
that copies the deferred:

    newdef = defer.Deferred()
    newdef.callbacks = list(self.deferred.callbacks)
    newdef.callback(False)


>> My third and final attempt was to tie the downloader in to my
>> application.  Every time data arrive, the downloader sends it to the
>> object that needs it.  This isn't a very general solution, but it is
>> very short and readable.
>
> This I actually like, and sounds a bit like what I would implement someday
> if I were to implement something.  Of course, it should be general-purpose
> so it works for other applications.  The idea I have is to allow resources
> to optionally handle these new data-arrived events.  It should be possible
> for a resource to signal that it wants data as it is received, perhaps by
> declaring that implements a new interface (eg, IStreamingRequestHandler or
> something) with a method that is called each time data is received in the
> request body.  Resources which don't implement this interface will get the
> old behavior of having the request body buffered and delivered all at once,
> but resources which want streaming can get it.
>
> Does that make sense?

That makes sense, but in the streaming case I really think that you need
to call back every time data comes in.

> If the above described approach sounds sensible and you want to take a
> crack at it, I'd really appreciate it.  The best thing to do with the
> result is attach it to a ticket in the tracker, probably #288
>
>  http://twistedmatrix.com/trac/ticket/288
>
> If not, I'd still like to see the code and make sure the ideas I have will
> also satisfy the use-case it represents.

I'm certainly willing to help, although I think we have some things to
clean up.

-- 
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-web/attachments/20080306/c95c47f9/attachment.pgp


More information about the Twisted-web mailing list