[Twisted-Python] Streaming HTTP

[Michael Schlenker]

Hi,

sounds like a good idea.

One topic: What should happen if the sender sends malformed data (e.g.
content-length: 100 and sending 200 bytes body). I would usually want to
know if the sender is broken instead of getting a responseComplete()
that claims everything is fine.

We simply subclassed twisted.web.server.Request and injected the needed
parts so we could get the data we wanted in an incremental way and set
some socket options (SO_RCVBUF) for multi-gigabyte file uploads.

So we took 'gotLength()' as a signal that headers are fully received and
also took 'write()', 'processingFailed()' to handle cleanup.

Basically we replace the self.content stream object with our own version
that does not buffer everything in memory.

We tried to do the 100-Continue stuff too, but it simply didn't work
with the Python stdlib httplib code (it just ignores those more or
less). So we either just terminate the connection forcefully (reset) or
read the full request and send it to /dev/null.

Michael

-- 
Michael Schlenker
Senior Software Engineer

CONTACT Software GmbH           Tel.:   +49 (421) 20153-80
Wiener Straße 1-3               Fax:    +49 (421) 20153-41
28359 Bremen
E-Mail: michael.schlenker at contact-software.com
http://www.contact-software.com/

Registered office: Bremen, Germany
Managing directors: Karl Heinz Zachries, Ralf Holtgrefe
Court of register: Amtsgericht Bremen HRB 1321