[Twisted-Python] Streaming HTTP

[Louis D. Burr]

Hi Cory,

> On Nov 13, 2015, at 6:36 AM, Cory Benfield <cory at lukasa.co.uk> wrote:
> 
> Folks,
> 
> # Problem Statement
> 
> Thanks for your feedback on my HTTP/2 questions. I’ve started work implementing a spike of a HTTP/2 protocol for twisted.web. I’m aiming to have something that works in at least some cases by the end of the day.
> 
> As part of my dive into twisted.web, I noticed something that surprised me: it seems to have no support for ‘streaming’ request bodies. By this I mean that the Request.requestReceived() method is not actually called until the complete request body has been received. This is a somewhat unexpected limitation for Twisted: why should I have to wait until the entire body has been uploaded to start doing things with it?
> 
> This problem is thrown into sharp relief with HTTP/2, which essentially always chunks the body, even if a content-length is provided. This means that it is now very easy to receive data in delimited chunks, which an implementation may want to have semantic meaning. However, the request is unable to access this data in this way. It also makes it impossible to use a HTTP/2 request/response pair as a long-running communication channel, as we cannot safely call requestReceived until the response is terminated (which also terminates the HTTP/2 stream).
> 
> Adi pointed me at a related issue, #6928[0], which itself points at what appears to be an issue tracking exactly this request. That issue is issue #288[1], which is 12 years old(!). This has clearly been a pain point for quite some time.
> 

I think it is worth noting that some of these issues were solved by twisted.web2, to the point that there were even adapters created that handled some of the differences between the existing web resource model and the new model in web2.

Of course, web2 did not succeed in replacing web, due in part to concerns over the introduction of a new flow-control mechanism (IStream and company).  Today though, we have https://github.com/twisted/tubes, which is being developed outside of twisted proper, but could perhaps be leveraged to overcome some of the previous concerns about introducing a mechanism different from IProducer/IConsumer.

I mention this because I think there is work within the web2 codebase that might prove useful in solving some of the issues being discussed here, in a way that would benefit twisted, and its users.


Thanks for reading,

L. Daniel Burr