[Twisted-web] streaming request (was: status of Twisted Web and Web 2)

[Jean-Paul Calderone]

On Thu, 6 Mar 2008 10:16:52 -0700, Andrew McNabb <amcnabb at mcnabbs.org> wrote:
>On Thu, Mar 06, 2008 at 11:42:15AM -0500, Christopher Armstrong wrote:
>>
>> I'm not sure how that applies. I really don't know how it would work.
>> Generally, getting access to a one-shot deferred is done by calling a
>> function which represents a request of some sort. This, on the other
>> hand, is the opposite: the request is being made of the resource, and
>> some data is being uploaded.
>
>Seriously, it would be the same.  You call a function which represents a
>request.  That function knows that the response might come incrementally
>rather than all at once, so it returns a multi-shot deferred.

This doesn't seem like it would be easier than having a non-Deferred based
callback (noun) solution.  Involving all the behavior of a Deferred seems
to mostly add complexity, not functionality.  For example, what do you need
errbacks here for?  What do you do if one of the "shots" of a MultiDeferred
ends in a Failure?  There's no way for the HTTP implementation to observe
this, so it would just continue to deliver data, even after an unrecoverable
unhandled error.  Useful?  Probably not.  And there would be other difficult
implementation issues.  For example, to present something that behaves even
remotely like a Deferred, it must be possible to add callbacks after a result
has been passed to the `callback´ method.  So MultiDeferred has to keep all
of its results around for its entire lifetime.  This doesn't seem useful in
most cases, but there's not really any way to avoid it.

So it might be the same as a single-shot Deferred, but since the use-case is
so radically different than a single-shot Deferred, being "the same" is now
a liability, not an advantage.

Perhaps there is a place for an abstraction for handling this sort of case
which is unrelated to Deferreds, but I haven't ever had a problem with the
"named callback (noun)" style before.  Admittedly, I have a much greater
familiarity with Twisted than the average user, so it's entirely possible
that there are difficulties I've become used to which, if fixed or
simplified, would greatly help people who have not developed as high a
tolerance.

Jean-Paul