Opened 8 years ago

Last modified 7 years ago

#5511 enhancement new

Streaming content of multipart/form-data requests server-side

Reported by: EDevil Owned by:
Priority: normal Milestone:
Component: web Keywords: streamming, multipart
Cc: EDevil Branch:
Author:

Description

Twisted is missing a mechanism to handle large multipart/form-data POST requests in a streaming fashion. The current twisted.web implementation loads the entire body into memory in order to parse it with the cgi module. Being able to access the body of the request as it is being read can also be useful in cases where the processing can start before the file is completely transferred.

Something in the lines of: http://debuggable.com/posts/streaming-file-uploads-with-node-js:4ac094b2-b6c8-4a7f-bd07-28accbdd56cb

Change History (5)

comment:1 Changed 8 years ago by Jean-Paul Calderone

Component: coreweb

I'm not sure how different this is from #288. Maybe it's just a thin layer on top of the functionality provided by #288 for parsing the request body into a more structured form representing a form post (including a stream for files that are part of the post).

comment:2 Changed 8 years ago by EDevil

Cc: EDevil added

Yes, that's why I filed a different issue. I think I can access the data being uploaded now by redefining some methods, as you said on the IRC channel. Being able to do this without messing around with Twisted internals would be nice, and I guess that is #288. But in the case of multipart requests there still is the issue of parsing these structures in a streaming fashion, which cgi.parse_multipart does not do and cgi.FieldStorage requires a file to read the data from.

comment:3 Changed 7 years ago by Jean-Paul Calderone

Relatedly, the multipart/form-data parser on Python 3 is broken (<http://bugs.python.org/issue12411>). This is perhaps another reason to implement our own.

comment:4 Changed 7 years ago by Itamar Turner-Trauring

I believe the Python bug above has been fixed. Also, email.FeedParser may be the way to go, given that it has incremental parsing.

comment:5 Changed 7 years ago by Jean-Paul Calderone

Or Parsley.

Note: See TracTickets for help on using tickets.