[Twisted-web] streaming request (was: status of Twisted Web and Web 2)

[Jean-Paul Calderone]

On Thu, 6 Mar 2008 07:15:53 -0700, Andrew McNabb <amcnabb at mcnabbs.org> wrote:
>On Wed, Mar 05, 2008 at 10:30:29PM -0500, Jean-Paul Calderone wrote:
>>
>> To clarify, this is about calling into application code in a web server
>> once each time some new bytes from a request body are received from the
>> client, right?
>
>Exactly.
>
>> Deferred.callback explicitly checks to see if you're giving it a Deferred
>> and raises the TypeError.  The idea here is that most of the time, you're
>> probably doing this by accident, not intentionally.  Clearly it wasn't an
>> accident in your case.
>
>I definitely was not doing it by accident. :)  Is this really a common
>mistake?

I have absolutely no statistics on the frequency of this error.  I suspect
that it isn't, but I may be wrong.  The check was added years ago, presumably
by someone who thought it was/would be.

>
>> The way to actually get this behavior is to chain the Deferreds
>> together.  Instead of a.callback(b), which I assume you was to send
>> the result of b to the callbacks of a when they became available, you
>> a.chainDeferred(b), which is really just a way to say
>>
>>  a.addCallbacks(b.callback, b.errback)
>>
>> However, I don't think this is generally the approach to take.
>
>This actually isn't equivalent to what I was trying to do.  Think of my
>approach this way.  As a user, you call:
>
>d = streaming_download(url)
>
>def got_some_data(value):
>    new_data, d = value
>    # add callback for the next time data get sent:
>    d.addCallback(got_some_data)
>    # do stuff with new_data
>
>d.addCallback(got_some_data)
>

Ah.  So the case that doesn't work is when the result of the Deferred is
really a new, perhaps totally unrelated, Deferred that someone wants.  I
can see how that would be useful.  I think I've actually encountered the
case before myself.  It might be worth complaining about this misfeature
on the other list (twisted-python).

>
>>> My second attempt was to create a new deferred each time data arrived
>>> and to copy the callback list from the previous deferred.  This worked,
>>> but it felt dirty, so I got rid of it.
>
>Actually, after looking at my code again, it turns out that what I am
>doing is actually a mix of attempt 2 and attempt 3.  This is the code
>that copies the deferred:
>
>    newdef = defer.Deferred()
>    newdef.callbacks = list(self.deferred.callbacks)
>    newdef.callback(False)
>

*cries*

You're not supposed to read or write the `callbacks´ attribute like that.

>
>>> My third and final attempt was to tie the downloader in to my
>>> application.  Every time data arrive, the downloader sends it to the
>>> object that needs it.  This isn't a very general solution, but it is
>>> very short and readable.
>>
>> This I actually like, and sounds a bit like what I would implement someday
>> if I were to implement something.  Of course, it should be general-purpose
>> so it works for other applications.  The idea I have is to allow resources
>> to optionally handle these new data-arrived events.  It should be possible
>> for a resource to signal that it wants data as it is received, perhaps by
>> declaring that implements a new interface (eg, IStreamingRequestHandler or
>> something) with a method that is called each time data is received in the
>> request body.  Resources which don't implement this interface will get the
>> old behavior of having the request body buffered and delivered all at once,
>> but resources which want streaming can get it.
>>
>> Does that make sense?
>
>That makes sense, but in the streaming case I really think that you need
>to call back every time data comes in.

Yes, definitely.  `IStreamingRequestHandler´ would have some method like
`someMoreRequestDataReceived´ and it would be called as many times as
necessary for a single request until the entire request body has been
delivered.

>
>> If the above described approach sounds sensible and you want to take a
>> crack at it, I'd really appreciate it.  The best thing to do with the
>> result is attach it to a ticket in the tracker, probably #288
>>
>>  http://twistedmatrix.com/trac/ticket/288
>>
>> If not, I'd still like to see the code and make sure the ideas I have will
>> also satisfy the use-case it represents.
>
>I'm certainly willing to help, although I think we have some things to
>clean up.
>

Jean-Paul