[Twisted-web] Large file uploads

[Julian Pietron]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

After some looks at twisted.web's http.py, I came to the conclusion,
that for my solution to work the way of receiving client's request has
to be modified quite much.
I think twisted does the body parsing when all data has been received
(i.e. is stored somewhere in memory). So it would be too late to hook in
at this point, because the whole file would've been loaded to memory
(or, rather, it wouldn't, because it simply wouldn't fit in). Instead,
the way to go would be to parse the Content-type header of the client's
request as early as possible to decide whether
a) do the parsing in the traditional, twisted.web way (which means to
wait until the request has been completely received and then parse it)
b) in case of multipart/form-data redirect the following input to an
instance of cgi.FieldStorage, which writes any files directly to disk
instead of loading their contents to memory, or, alternatively, write an
own FieldStorage-like class (but why should one re-implement already
existing and well-tested classes?)

The only question that remains is that, while it would (imho) be quiet
easy to watch out for an multipart/form-data header, where do I generate
the input stream from that FieldStorage expects? In cherrypy, this is
quiet simple, because it's not async as twisted is but uses a more
traditional approach to sockets, so that an input stream is easily
available. But as twisted is async and dataReceived() is called on
received data, there is no input stream available to direct to
FieldStorage, or am I wrong in this conclusion? If I'm not, is there any
way to supply FieldStorage with an input stream without pre-reading the
whole request body to memory?

Thanks,
Julian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJMSgsnAAoJENidYKvYQHlQhNgQAJk4u3VklMZeJkHvCn/H+mHv
i6TDwkGAXEv5fhPxbSVNb+5zDQUS1w305Zx9/oQeJldViKzGN82p4x2D5doapzOI
OD7Nm8wDo/giFByjLwXffxvpFKcz/fyKgS/eK8w6gj0MJQ9ZIqpwEXj9L3OvmFgx
VUaVO3hDv4BRIxLCvG1dZYmdvqnY0W8/z1Bv472ZU48SoK+NhF+zNhhq3VPFDwOv
J56HWrWR7DXjoBZvgxr6r39kmjpblkoknppAXhgEMMPmsSsKe3tG4QVuFUkvqjKL
4IajlSo+Lu9bqAwHhZBw91XfFJhWtflWxQQjgoCWAa3TeNj97WrDGI3y4MRCnDyt
36i4DRK1FMqmnUcxEXMW0C0pviU4caQASmQ+74RsgGOsgCFOtznZOEBk4RN+cLLk
6ONPHpD3q0mHhrzFcIQTgebocMyjnmK+I8AuPirwrVwF1plsKxozNTmHPPJKR1F3
hdWWk0z7ywOlkfpgBpO8sbCmp0bNbryCpCXLG8Qq3SqKbFW8nfwC/SYWr0P1nM6t
VBTAxw31HfH0897xXK/wdnGIM7glBkypagcVp+YcyNJx/QN33zEA7M2Gn1h4sWpg
o+tpTfbI201wrUWooM8U0srGa0fLB3ftjXAFfwOUwJVUVmP47hZLDisPaHrzdmZU
dpTV/NcYhuqSnN+PgflF
=z4LY
-----END PGP SIGNATURE-----