[Twisted-web] Large file uploads

[Julian Pietron]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 24.07.2010 05:50, schrieb exarkun at twistedmatrix.com:
> On 23 Jul, 08:34 pm, julian at whisper-net.de wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Am 16.07.2010 17:33, schrieb exarkun at twistedmatrix.com:
>>>
>>> You can override `Request.gotLength` to inspect the request and select
>>> an appropriate destination for the request body.
>>>
>>> You can convince Site to use your custom request by setting it as the
>>> `requestFactory` attribute.
>>>
>>> Jean-Paul
>>
>> Hi,
>>
>> thanks for your reply. In meantime I tried cherrypy, which does its job
>> very good but has some flaws compared to my twisted applications when 
>> it
>> comes to performance and resource usage. So I'm still interested in
>> solving my problem by using twisted.
>> The problem is as follows: The file which shall be uploaded is sent via
>> a HTML form (encoded as multipart/form-data), and I'd like to use
>> Python's FieldStorage to parse the request body because it handles the
>> file matters very well. You can tell cherrypy, that for a certain
>> request uri cherrypy should not parse the request body but instead pass
>> it on unparsed to the handler method.
>> Is there a simple way of configuring twisted.web in this way to?
>> So that e.g. if request uri /do-upload-form is issued, twisted.web does
>> not try to parse the request body on its own and generate the args dict
>> to pass on to the render_POST method, but instead uses FieldStorage (or
>> better: my own subclass of FieldStorage, which employs a NamedTempFile
>> instead of an anonymous tempfile, which allows me to simply rename it
>> after it has passed some sanity checks) for parsing the body?
> 
> This will be easier once #288 is resolved.  Until then, you just have to 
> hack around with the Request to change the behavior at a very low level.
> 
> Jean-Paul
> 

If I've unterstood the source code of http.py, class Request right, the
methods to override would be:
1. handleContentChunk, which is called after some data has been received
by the transfer decoder. Something that behaves like cgi.FieldStorage,
but in an async way, should be installed here / instead of the StringIO
used for self.content
2. requestReceived, which has nothing to do anymore besides of calling
self.process

The only thing that misses now is request's command (GET, POST, HEAD
etc.) and request path. Is there a way to install these in the Request
instance before allContentReceived is called other than overriding
HTTPChannel's lineReceived?

Thanks,
Julian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJMSt1YAAoJENidYKvYQHlQt5UP+waxLHEVCHQOOY7saeECwEEZ
QsxZpsEyHKd8imcpfrzIgQaooj5OlcFLpAWkaaGFOhUDxq6RmXrLL60/awCTSiCI
Xv1EsNJiwml5/uemhny4BCUq1aBcM0qcMg6BAZG8jwDy2nAVgyX3kall2a+17ybr
aBnssceBHFJm2I7o/yV9ZkrzIPtOabZKqEAyBlE9pqYA1SHZfKhuUZw+BQMpdGdV
HnE+58YRk39B1sTi5GQc5Qed3ynTxeE8tLTHgvi/99QMs+pGQP7kmFbq5KYch8ks
IGVBYJCZXoSZ6aow6zkYSZ8/hmiI5bS3OAnzVSkcOz122UlCqhicSQ7aUUJXrKKs
Er9R4ETNbCYAkZqw+8XofSgcawuP3XxDBfChPeDAC7G1bjnVEO7vEAeUyl+wQ9AO
UXnQde25HAIz88ouXo50Au7Q0Wx/qpnC7Xnl2u6Bbykj0c1UNg1146ZyFCXx3L7f
OT7kQgAIGm3cbCtVU1RLKegsXJD318xEEAHEUoAOjLzbHbuM7cLy6lP5udz/U4QF
pcbpEahNesMKiD7qCGrFzp7nEHu2A272iSN3rbOq0YQILCMhe9uBFqwP1KPsPcMQ
2Or1/n0meu00kZqrFy2t7SsPg3c2754UwC9kLwWXrsP/XoP3HG+YeBFe54sowB2S
lEmLCSwtNlEHH0dy4rmr
=IJw9
-----END PGP SIGNATURE-----