Opened 15 years ago
Closed 10 years ago
#1873 defect closed wontfix (wontfix)
Large File Uploads Fail
Reported by: | lennyg | Owned by: | |
---|---|---|---|
Priority: | normal | Milestone: | Web2-Gold-Master |
Component: | web2 | Keywords: | |
Cc: | Branch: | ||
Author: |
Description (last modified by )
resource.PostableResource doesn't seem able to handle file uploads much larger than 9M. At 10M, the render() method is never entered. Here is a code snippet that can be used to reproduce the error:
import os from twisted.internet import reactor from twisted.web2 import server, http, resource, channel from twisted.web2 import http_headers, responsecode from twisted.web2 import iweb, stream SAVEDIR = "/tmp" READSIZE=8192 class UploadFile(resource.PostableResource): def render(self, ctx): request = iweb.IRequest(ctx) filename = request.files['filename'][0][0] file = request.files['filename'][0][2] filestream = stream.FileStream(file) dest = os.path.join(SAVEDIR,filename) destfile = os.fdopen(os.open(dest, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0644), 'w', 0) stream.readIntoFile(filestream, destfile) msg = "saved %s to %s" % (filename, dest) print msg return http.Response(stream="%s" % msg) class Toplevel(resource.Resource): addSlash = True def render(self, ctx): return http.Response(responsecode.OK, {'content-type': http_headers.MimeType('text', 'html')}, "Hello") child_uploadfile = UploadFile() if __name__ == "__main__": site = server.Site(Toplevel()) reactor.listenTCP(1080, channel.HTTPFactory(site)) reactor.run() And here is an html form that can be used to test, with files of varying sizes: <html> <form action="http://localhost:1080/uploadfile" enctype="multipart/form-data" method="post"> filename: <input type="file" name="filename"> <input type="submit" value="submit"> </form> </html>
Programmatic clients (written in e.g. httplib) also get blocked upon writing only a few hunder KB if the file is larger than 10MB.
Attachments (2)
Change History (16)
Changed 15 years ago by
Attachment: | uploadserver.py added |
---|
comment:1 Changed 15 years ago by
Description: | modified (diff) |
---|
Modified submission for proper formatting.
comment:2 Changed 15 years ago by
Component: | core → web2 |
---|
Sorry -- didn't realize I could "Attach" (button didn't show up on entry screen) nor that inlined code would get all wonky.
comment:3 Changed 15 years ago by
James Y. Knight wrote (http://twistedmatrix.com/pipermail/twisted-web/2006-June/002733.html):
The failing with > 10MB is intentional, but the hanging is not. It's supposed to return an error response to the client. In fileupload.py: 233, parseMultipartFormData has arguments which limit the number of fields and the size of the uploaded data. The default limit for size is 10MB.
Unfortunately, it looks like there is no way to pass in different limits to fileupload.parseMultipartFormData from where it's called in server.parsePOSTData.
So there's two bugs here: 1) hangs instead of returning a response. 2) limit is not configurable.
comment:4 Changed 15 years ago by
Initial diagnostics show that it is indeed returning an error response but that the client doesn't get it. Perhaps because it's still blockingly trying to upload the rest of the file. It seems as though the server has paused reading the input (thus causing the client to block) rather than ignoring the rest.
comment:5 Changed 15 years ago by
Yes, that is exactly it. The server is supposed to close the connection when responding to a request before the complete request has been read, which it currently doesn't do.
comment:6 Changed 15 years ago by
Owner: | changed from David Reid to jknight |
---|
comment:7 Changed 14 years ago by
Milestone: | → Web2-Gold-Master |
---|
comment:8 follow-up: 9 Changed 14 years ago by
Owner: | changed from jknight to therve |
---|
comment:9 Changed 14 years ago by
Replying to therve:
the previously included code isn't right. the following demonstrates the problem properly:
#!/usr/bin/python import os from twisted.internet import reactor from twisted.web2 import server, http, resource, channel, stream from twisted.web2 import http_headers, responsecode from twisted.web2 import iweb SAVEDIR = "/tmp" class UploadFile(resource.PostableResource): def render(self, ctx): request = iweb.IRequest(ctx) filename = request.files['filename'][0][0] file = request.files['filename'][0][2] filestream = stream.FileStream(file) dest = os.path.join(SAVEDIR,filename) destfile = os.fdopen(os.open(dest, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0644), 'w', 0) d = stream.readIntoFile(filestream, destfile) d.addBoth(self.cleanup, destfile, file, "saving %s to %s" % (filename, dest)) return d def cleanup(self, r, destfile, file, msg): destfile.close() file.close() if r: print "error: %s" % r msg = "%s failed" % msg else: msg = "%s succeeded" % msg print msg return http.Response(stream="%s" % msg) class Toplevel(resource.Resource): addSlash = True def render(self, ctx): return http.Response(responsecode.OK, {'content-type': http_headers.MimeType('text', 'html')}, "Hello") child_uploadfile = UploadFile() if __name__ == "__main__": site = server.Site(Toplevel()) reactor.listenTCP(1080, channel.HTTPFactory(site)) reactor.run()
I'm still getting a hang with files >= 10MB, using 0.2.0+svn20070403, but very glad to see this targetted for GoldMaster!
comment:10 Changed 13 years ago by
There is some stuff in the large-fileupload-1873 branch. The configuration of the limit is somewhat done. But the real bug mentioned here is not, because I don't manage to create an unit test for it.
comment:11 Changed 13 years ago by
comment:12 Changed 11 years ago by
Owner: | therve deleted |
---|
comment:13 Changed 10 years ago by
Resolution: | → wontfix |
---|---|
Status: | new → closed |
Not going to fix this, see #4821. Twisted Web doesn't have a similar problem with large uploads, so not retargeting at the web component either.
uploatserver.py