[Twisted-web] streaming request (was: status of Twisted Web and Web 2)

[Jean-Paul Calderone]

On Wed, 5 Mar 2008 20:09:52 -0700, Andrew McNabb <amcnabb at mcnabbs.org> wrote:
>On Wed, Mar 05, 2008 at 09:13:44PM -0500, Jean-Paul Calderone wrote:
>> On Wed, 5 Mar 2008 18:52:50 -0700, Andrew McNabb <amcnabb at mcnabbs.org> wrote:
>>>
>>> I've written some code that uses Twisted Web, and it seemed a little
>>> basic.  For example, I had to make some modifications in order to
>>> receive large files in progress instead of waiting until they were done
>>> downloading.
>>
>> Does this code have tests?  Is it compatible with "normal" twisted.web
>> usage?  Can I have the code?
>
>My answer to all three questions is "sort of."  If you (or others) could
>help me resolve a problem I ran into, I could change all of those to a
>"yes."
>
>The problem that I ran into is that with streaming requests, I would
>like to be able to call back multiple times.  It makes sense that a
>normal deferred can only be called back once.  I couldn't find anything
>in Twisted that you might call a "multi-deferred" (to make it simple to
>callback multiple times).  It's been a while, so let me see if I
>remember what I tried.

To clarify, this is about calling into application code in a web server
once each time some new bytes from a request body are received from the
client, right?

>
>My first attempt was to callback with a new deferred as the callback
>value.  Then the callback function could add itself to the new deferred
>and get called again the next time data arrive.  I thought this was a
>clean approach, but it turns out that if you try to call back with a new
>deferred, there's a TypeError.  I couldn't figure out why it didn't like
>this.

Deferred.callback explicitly checks to see if you're giving it a Deferred
and raises the TypeError.  The idea here is that most of the time, you're
probably doing this by accident, not intentionally.  Clearly it wasn't an
accident in your case.  The way to actually get this behavior is to chain
the Deferreds together.  Instead of a.callback(b), which I assume you was
to send the result of b to the callbacks of a when they became available,
you a.chainDeferred(b), which is really just a way to say

  a.addCallbacks(b.callback, b.errback)

However, I don't think this is generally the approach to take.

>
>My second attempt was to create a new deferred each time data arrived
>and to copy the callback list from the previous deferred.  This worked,
>but it felt dirty, so I got rid of it.
>
>My third and final attempt was to tie the downloader in to my
>application.  Every time data arrive, the downloader sends it to the
>object that needs it.  This isn't a very general solution, but it is
>very short and readable.

This I actually like, and sounds a bit like what I would implement someday
if I were to implement something.  Of course, it should be general-purpose
so it works for other applications.  The idea I have is to allow resources
to optionally handle these new data-arrived events.  It should be possible
for a resource to signal that it wants data as it is received, perhaps by
declaring that implements a new interface (eg, IStreamingRequestHandler or
something) with a method that is called each time data is received in the
request body.  Resources which don't implement this interface will get the
old behavior of having the request body buffered and delivered all at once,
but resources which want streaming can get it.

Does that make sense?

>
>Anyway, if you know of a better way to deal with this issue, I would be
>happy to modify my code for it.  If you're happy with what I've got, I
>would be happy to email it to you, but I wouldn't recommend adding it to
>Twisted Web until it's more general.

If the above described approach sounds sensible and you want to take a
crack at it, I'd really appreciate it.  The best thing to do with the
result is attach it to a ticket in the tracker, probably #288

  http://twistedmatrix.com/trac/ticket/288

If not, I'd still like to see the code and make sure the ideas I have will
also satisfy the use-case it represents.

Thanks!

Jean-Paul