[Twisted-Python] Implement HTTP request acception / rejection as defined in RFC 2616 - 8.2.3

[Adi Roiban]

Hi,

As requested during review I am moving this to mailing list for discussions.

The initial ticket is here : https://twistedmatrix.com/trac/ticket/6928

-------

The current Twisted implementation of RFC 2616 - 8.2.3 is very simple and
Twisted will always reply with continue to Accept: 100-continue requests.

I am working at web application based on Twisted web so I did a quick
security check for current twisted web code.

The major problem is that when reading POST data, requests under 1M are
kept in memory, the other being stored in temporary file... but for
form-mulipart request the whole data is loaded again in memory for parsing.
Now, if the client initially reports that content is less than 1M, but then
continue to send more than 1M, twisted.web will continue to keep all data
in memory.

To address the problem of unwanted big post requests I first start with
implementing support for well behaved HTTP/1.1 clients.

In current code, Resource objects are called only after whole request data
was received and processed so a resource can not decide whether or not it
accepts the request, just by looking at headers.

Also the Request object is called after headers are received but with calls
like parseCookie or gotLength. So there is no API for Request to accept or
reject a request based on headers.

--------

To solve this problem, I add a new `headersReceived` API call on Request
and Resource so that Requests and Resources can decided if they want to
accept the request.

The biggest change required to implement this API is that URL traversal can
now begin before a resource has received all its content.
If a resource needs full request content before making a traversal
decistion, then this API does not work.

This is why this is an optional API not enforced on all resources.

For now the ticket only has demo code as I wanted to highlight the API
changes.

I have already implemented this code in my project and wrote full test for
these changes.

If this kind of API is accepted in Twisted I can port the code and tests,
otherwise I can try to work on a different implementation.

Thanks!

-- 
Adi Roiban
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20140221/99fc1b81/attachment.html>