Opened 11 years ago
Closed 7 years ago
#1873 defect closed wontfix (wontfix)
Large File Uploads Fail
| Reported by: | lennyg | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | Web2-Gold-Master |
| Component: | web2 | Keywords: | |
| Cc: | Branch: | ||
| Author: |
Description (last modified by )
resource.PostableResource doesn't seem able to handle file uploads much larger than 9M. At 10M, the render() method is never entered. Here is a code snippet that can be used to reproduce the error:
import os
from twisted.internet import reactor
from twisted.web2 import server, http, resource, channel
from twisted.web2 import http_headers, responsecode
from twisted.web2 import iweb, stream
SAVEDIR = "/tmp"
READSIZE=8192
class UploadFile(resource.PostableResource):
def render(self, ctx):
request = iweb.IRequest(ctx)
filename = request.files['filename'][0][0]
file = request.files['filename'][0][2]
filestream = stream.FileStream(file)
dest = os.path.join(SAVEDIR,filename)
destfile = os.fdopen(os.open(dest,
os.O_WRONLY | os.O_CREAT | os.O_EXCL,
0644), 'w', 0)
stream.readIntoFile(filestream, destfile)
msg = "saved %s to %s" % (filename, dest)
print msg
return http.Response(stream="%s" % msg)
class Toplevel(resource.Resource):
addSlash = True
def render(self, ctx):
return http.Response(responsecode.OK,
{'content-type': http_headers.MimeType('text', 'html')},
"Hello")
child_uploadfile = UploadFile()
if __name__ == "__main__":
site = server.Site(Toplevel())
reactor.listenTCP(1080, channel.HTTPFactory(site))
reactor.run()
And here is an html form that can be used to test, with files of varying sizes:
<html>
<form action="http://localhost:1080/uploadfile"
enctype="multipart/form-data" method="post">
filename: <input type="file" name="filename">
<input type="submit" value="submit">
</form>
</html>
Programmatic clients (written in e.g. httplib) also get blocked upon writing only a few hunder KB if the file is larger than 10MB.
Attachments (2)
Change History (16)
Changed 11 years ago by
| Attachment: | uploadserver.py added |
|---|
comment:1 Changed 11 years ago by
| Description: | modified (diff) |
|---|
Modified submission for proper formatting.
comment:2 Changed 11 years ago by
| Component: | core → web2 |
|---|
Sorry -- didn't realize I could "Attach" (button didn't show up on entry screen) nor that inlined code would get all wonky.
comment:3 Changed 11 years ago by
James Y. Knight wrote (http://twistedmatrix.com/pipermail/twisted-web/2006-June/002733.html):
The failing with > 10MB is intentional, but the hanging is not. It's supposed to return an error response to the client. In fileupload.py: 233, parseMultipartFormData has arguments which limit the number of fields and the size of the uploaded data. The default limit for size is 10MB.
Unfortunately, it looks like there is no way to pass in different limits to fileupload.parseMultipartFormData from where it's called in server.parsePOSTData.
So there's two bugs here: 1) hangs instead of returning a response. 2) limit is not configurable.
comment:4 Changed 11 years ago by
Initial diagnostics show that it is indeed returning an error response but that the client doesn't get it. Perhaps because it's still blockingly trying to upload the rest of the file. It seems as though the server has paused reading the input (thus causing the client to block) rather than ignoring the rest.
comment:5 Changed 11 years ago by
Yes, that is exactly it. The server is supposed to close the connection when responding to a request before the complete request has been read, which it currently doesn't do.
comment:6 Changed 11 years ago by
| Owner: | changed from David Reid to jknight |
|---|
comment:7 Changed 11 years ago by
| Milestone: | → Web2-Gold-Master |
|---|
comment:8 follow-up: 9 Changed 10 years ago by
| Owner: | changed from jknight to therve |
|---|
comment:9 Changed 10 years ago by
Replying to therve:
the previously included code isn't right. the following demonstrates the problem properly:
#!/usr/bin/python
import os
from twisted.internet import reactor
from twisted.web2 import server, http, resource, channel, stream
from twisted.web2 import http_headers, responsecode
from twisted.web2 import iweb
SAVEDIR = "/tmp"
class UploadFile(resource.PostableResource):
def render(self, ctx):
request = iweb.IRequest(ctx)
filename = request.files['filename'][0][0]
file = request.files['filename'][0][2]
filestream = stream.FileStream(file)
dest = os.path.join(SAVEDIR,filename)
destfile = os.fdopen(os.open(dest,
os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0644), 'w', 0)
d = stream.readIntoFile(filestream, destfile)
d.addBoth(self.cleanup, destfile, file,
"saving %s to %s" % (filename, dest))
return d
def cleanup(self, r, destfile, file, msg):
destfile.close()
file.close()
if r:
print "error: %s" % r
msg = "%s failed" % msg
else:
msg = "%s succeeded" % msg
print msg
return http.Response(stream="%s" % msg)
class Toplevel(resource.Resource):
addSlash = True
def render(self, ctx):
return http.Response(responsecode.OK,
{'content-type': http_headers.MimeType('text', 'html')},
"Hello")
child_uploadfile = UploadFile()
if __name__ == "__main__":
site = server.Site(Toplevel())
reactor.listenTCP(1080, channel.HTTPFactory(site))
reactor.run()
I'm still getting a hang with files >= 10MB, using 0.2.0+svn20070403, but very glad to see this targetted for GoldMaster!
comment:10 Changed 10 years ago by
There is some stuff in the large-fileupload-1873 branch. The configuration of the limit is somewhat done. But the real bug mentioned here is not, because I don't manage to create an unit test for it.
comment:11 Changed 10 years ago by
comment:12 Changed 8 years ago by
| Owner: | therve deleted |
|---|
comment:13 Changed 7 years ago by
| Resolution: | → wontfix |
|---|---|
| Status: | new → closed |
Not going to fix this, see #4821. Twisted Web doesn't have a similar problem with large uploads, so not retargeting at the web component either.
uploatserver.py