[Twisted-Python] 100-continue and rejecting connection after headers are received

[Adi Roiban]

I am moving this to a dedicated ticket

I start working on 100-Continue here
https://twistedmatrix.com/trac/ticket/6928 and was asked to follow up
on the mailinglist ... which due to slow review progress I gave up..
so I am doing it now.


On 19 March 2015 at 15:53, HawkOwl <hawkowl at atleastfornow.net> wrote:
>
>> On 19 Mar 2015, at 23:26, Adi Roiban <adi at roiban.ro> wrote:
>>
>> Hi,
>>
>> My usecase is this:
>>
>> When all headers of an HTTP request are received I want to ask a
>> remote service if the
>> request should be accepted and what to do with it.
>>
>> Before I got the response for the remote service, I don't want to
>> receive/read HTTP data.
>> The response is returned via an deferred.
>>
>> The HTTP data might be a 10G file which might be rejected.. for
>> example due to quota limits.
>>
>> Here is a gist with a short example and my quick and dirty fix:
>> https://gist.github.com/adiroiban/621efeab47662a04cb20
>>
>> ------
>>
>> Do you think this is a valid use case and that  someone would be
>> willing to review a patch to add such a feature?
>
> Ah yes, the 100-Continue stuff! I mentioned it briefly in my talk at PyCon AU.
>
> It would be a useful feature, and I looked into it a little bit, but I'm not entirely sure how it would be accessible to user code. Most web server frameworks (Django, Klein, Flask, et al) work on a "here's a full request, send a full response" (only Twisted/Klein can do more than send a whole response without hacks :) ) and fitting it into that sort of framework would be a bit confusing.
>
> So, my question is, because this functionality is desirable, how would it be exposed to the user? Maybe if it did traversal prior to the whole request being completed, and called something like headers_POST() or headers_GET() similar to render_GET/POST/etc, which would give the user access to these headers. A Deferred firing with True unblocks the producer, a Deferred firing with False kills off the request.

I already have 100-continue in production ... and I also have a code
which rejects a request after headers were received even if
100-continue was not used.

When all headers are received I am traversing the url and inform the
resource about the headers... before content is received.

I have a special resource which has a new headersReceived method. This
method should return http._CONTINUE as otherwise the request is
rejected.

This only work for resource traversals which don't depend on the request body.
Exarkun made an excellent comment here
https://twistedmatrix.com/trac/ticket/6928#comment:5

The new resource interface is simple.

Here is a dump from my current code
https://gist.github.com/adiroiban/1a515a9354a8168a0275

Here is a diff on Twisted
https://github.com/twisted/twisted/compare/trunk...chevah:6928-http-100-accept

Please review the interface and let me know if it make sense to you.
I am happy to continue working on this as long as others are
interested in reviewing the patch.

Thanks!

-- 
Adi Roiban