[Twisted-web] streaming request (was: status of Twisted Web and Web 2)

[Jean-Paul Calderone]

On Thu, 6 Mar 2008 10:33:39 -0700, Andrew McNabb <amcnabb at mcnabbs.org> wrote:
>On Thu, Mar 06, 2008 at 12:22:04PM -0500, Christopher Armstrong wrote:
>>
>> Ok, so the new interface is request.getStreamingData(). So the
>> explanation of the new feature is either:
>>
>> "Implement dataReceived and call getStreamingData on the request and
>> add self.dataReceived as a callback"
>>
>> or
>>
>> "declare implements(IStreamingRequestHandler) and implement dataReceived".
>
>Now we understand each other.
>
>> Is the former really simpler? Does it really offer any benefit?
>
>I am opposed to creating a brand new IStreamingRequestHandler.  That's
>just more conceptual overhead.  However, if it were to use something
>that already exists, like IProtocol as Glyph suggested, I would be more
>accepting.

FWIW, IConsumer is a better match here than IProtocol (that's a more or
less irrelevant detail, though).

>
>In that case the difference is that the former has an event-driven feel
>and the latter doesn't.  Which is better?  It depends on what you're
>doing.  If you have multiple places that need notification that data
>have arrived (as is the case in my situation), there are some definite
>advantages to the event-driven approach.  However, the latter is
>probably a little faster.
>

They're equally "event-driven", so I think the feeling you have is only
due to familiarity or personal preference. :)

That said, here's where I see IStreamingRequestHandler as better than
re-using an existing interface like IProtocol or IConsumer:

  * IConsumer is a very general interface.  It provides no clues about its
    interaction with the HTTP protocol implementation.  There's no way to
    deduce that if an IResource is also an IConsumer that uploads will be
    streamed to it instead of delivered all at once at the end.  On the
    other hand, IStreamingRequestHandler can have documentation which
    describes its purpose and behavior.

  * IConsumer has existing semantics.  There's no reason an existing
    IResource implementation might not already be an IConsumer as well,
    for a completely unrelated reason.  If it suddenly starts to receive
    streaming upload data, it will almost certainly break.  On the other
    hand, IStreamingRequestHandler is a new interface, so it cannot be
    misinterpreted.

  * IConsumer isn't actually sufficiently expressive for this case.  It
    would be quite useful if the resource had a chance to look at the
    request headers before streaming begins (okay, I didn't bring this up
    earlier, so it may have seemed like IConsumer was sufficient).  On the
    other hand, IStreamingRequestHandler can have a method which takes the
    request as an argument.

It may be the case that the right method for IStreamingRequestHandler to
have, though, is one which takes the request and returns an IConsumer to
which the request body will be streamed.

Jean-Paul