[Twisted-web] streaming request (was: status of Twisted Web
and Web 2)
Andrew McNabb
amcnabb at mcnabbs.org
Thu Mar 6 09:15:53 EST 2008
On Wed, Mar 05, 2008 at 10:30:29PM -0500, Jean-Paul Calderone wrote:
>
> To clarify, this is about calling into application code in a web server
> once each time some new bytes from a request body are received from the
> client, right?
Exactly.
> Deferred.callback explicitly checks to see if you're giving it a Deferred
> and raises the TypeError. The idea here is that most of the time, you're
> probably doing this by accident, not intentionally. Clearly it wasn't an
> accident in your case.
I definitely was not doing it by accident. :) Is this really a common
mistake?
> The way to actually get this behavior is to chain the Deferreds
> together. Instead of a.callback(b), which I assume you was to send
> the result of b to the callbacks of a when they became available, you
> a.chainDeferred(b), which is really just a way to say
>
> a.addCallbacks(b.callback, b.errback)
>
> However, I don't think this is generally the approach to take.
This actually isn't equivalent to what I was trying to do. Think of my
approach this way. As a user, you call:
d = streaming_download(url)
def got_some_data(value):
new_data, d = value
# add callback for the next time data get sent:
d.addCallback(got_some_data)
# do stuff with new_data
d.addCallback(got_some_data)
>> My second attempt was to create a new deferred each time data arrived
>> and to copy the callback list from the previous deferred. This worked,
>> but it felt dirty, so I got rid of it.
Actually, after looking at my code again, it turns out that what I am
doing is actually a mix of attempt 2 and attempt 3. This is the code
that copies the deferred:
newdef = defer.Deferred()
newdef.callbacks = list(self.deferred.callbacks)
newdef.callback(False)
>> My third and final attempt was to tie the downloader in to my
>> application. Every time data arrive, the downloader sends it to the
>> object that needs it. This isn't a very general solution, but it is
>> very short and readable.
>
> This I actually like, and sounds a bit like what I would implement someday
> if I were to implement something. Of course, it should be general-purpose
> so it works for other applications. The idea I have is to allow resources
> to optionally handle these new data-arrived events. It should be possible
> for a resource to signal that it wants data as it is received, perhaps by
> declaring that implements a new interface (eg, IStreamingRequestHandler or
> something) with a method that is called each time data is received in the
> request body. Resources which don't implement this interface will get the
> old behavior of having the request body buffered and delivered all at once,
> but resources which want streaming can get it.
>
> Does that make sense?
That makes sense, but in the streaming case I really think that you need
to call back every time data comes in.
> If the above described approach sounds sensible and you want to take a
> crack at it, I'd really appreciate it. The best thing to do with the
> result is attach it to a ticket in the tracker, probably #288
>
> http://twistedmatrix.com/trac/ticket/288
>
> If not, I'd still like to see the code and make sure the ideas I have will
> also satisfy the use-case it represents.
I'm certainly willing to help, although I think we have some things to
clean up.
--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-web/attachments/20080306/c95c47f9/attachment.pgp
More information about the Twisted-web
mailing list