Opened 10 years ago
Closed 6 years ago
#1873 defect closed wontfix (wontfix)
Large File Uploads Fail
| Reported by: | lennyg | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | Web2-Gold-Master |
| Component: | web2 | Keywords: | |
| Cc: | Branch: | ||
| Author: |
Description (last modified by jknight)
resource.PostableResource doesn't seem able to handle file uploads much larger than 9M. At 10M, the render() method is never entered. Here is a code snippet that can be used to reproduce the error:
import os
from twisted.internet import reactor
from twisted.web2 import server, http, resource, channel
from twisted.web2 import http_headers, responsecode
from twisted.web2 import iweb, stream
SAVEDIR = "/tmp"
READSIZE=8192
class UploadFile(resource.PostableResource):
def render(self, ctx):
request = iweb.IRequest(ctx)
filename = request.files['filename'][0][0]
file = request.files['filename'][0][2]
filestream = stream.FileStream(file)
dest = os.path.join(SAVEDIR,filename)
destfile = os.fdopen(os.open(dest,
os.O_WRONLY | os.O_CREAT | os.O_EXCL,
0644), 'w', 0)
stream.readIntoFile(filestream, destfile)
msg = "saved %s to %s" % (filename, dest)
print msg
return http.Response(stream="%s" % msg)
class Toplevel(resource.Resource):
addSlash = True
def render(self, ctx):
return http.Response(responsecode.OK,
{'content-type': http_headers.MimeType('text', 'html')},
"Hello")
child_uploadfile = UploadFile()
if __name__ == "__main__":
site = server.Site(Toplevel())
reactor.listenTCP(1080, channel.HTTPFactory(site))
reactor.run()
And here is an html form that can be used to test, with files of varying sizes:
<html>
<form action="http://localhost:1080/uploadfile"
enctype="multipart/form-data" method="post">
filename: <input type="file" name="filename">
<input type="submit" value="submit">
</form>
</html>
Programmatic clients (written in e.g. httplib) also get blocked upon writing only a few hunder KB if the file is larger than 10MB.
Attachments (2)
Change History (16)
Changed 10 years ago by lennyg
comment:1 Changed 10 years ago by jknight
- Description modified (diff)
Modified submission for proper formatting.
comment:2 Changed 10 years ago by lennyg
- Component changed from core to web2
Sorry -- didn't realize I could "Attach" (button didn't show up on entry screen) nor that inlined code would get all wonky.
comment:3 Changed 10 years ago by lennyg
James Y. Knight wrote (http://twistedmatrix.com/pipermail/twisted-web/2006-June/002733.html):
The failing with > 10MB is intentional, but the hanging is not. It's supposed to return an error response to the client. In fileupload.py: 233, parseMultipartFormData has arguments which limit the number of fields and the size of the uploaded data. The default limit for size is 10MB.
Unfortunately, it looks like there is no way to pass in different limits to fileupload.parseMultipartFormData from where it's called in server.parsePOSTData.
So there's two bugs here: 1) hangs instead of returning a response. 2) limit is not configurable.
comment:4 Changed 10 years ago by jknight
Initial diagnostics show that it is indeed returning an error response but that the client doesn't get it. Perhaps because it's still blockingly trying to upload the rest of the file. It seems as though the server has paused reading the input (thus causing the client to block) rather than ignoring the rest.
comment:5 Changed 10 years ago by jknight
Yes, that is exactly it. The server is supposed to close the connection when responding to a request before the complete request has been read, which it currently doesn't do.
comment:6 Changed 10 years ago by dreid
- Owner changed from dreid to jknight
comment:7 Changed 10 years ago by glyph
- Milestone set to Web2-Gold-Master
comment:8 follow-up: ↓ 9 Changed 9 years ago by therve
- Owner changed from jknight to therve
comment:9 in reply to: ↑ 8 Changed 9 years ago by lennyg
Replying to therve:
the previously included code isn't right. the following demonstrates the problem properly:
#!/usr/bin/python
import os
from twisted.internet import reactor
from twisted.web2 import server, http, resource, channel, stream
from twisted.web2 import http_headers, responsecode
from twisted.web2 import iweb
SAVEDIR = "/tmp"
class UploadFile(resource.PostableResource):
def render(self, ctx):
request = iweb.IRequest(ctx)
filename = request.files['filename'][0][0]
file = request.files['filename'][0][2]
filestream = stream.FileStream(file)
dest = os.path.join(SAVEDIR,filename)
destfile = os.fdopen(os.open(dest,
os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0644), 'w', 0)
d = stream.readIntoFile(filestream, destfile)
d.addBoth(self.cleanup, destfile, file,
"saving %s to %s" % (filename, dest))
return d
def cleanup(self, r, destfile, file, msg):
destfile.close()
file.close()
if r:
print "error: %s" % r
msg = "%s failed" % msg
else:
msg = "%s succeeded" % msg
print msg
return http.Response(stream="%s" % msg)
class Toplevel(resource.Resource):
addSlash = True
def render(self, ctx):
return http.Response(responsecode.OK,
{'content-type': http_headers.MimeType('text', 'html')},
"Hello")
child_uploadfile = UploadFile()
if __name__ == "__main__":
site = server.Site(Toplevel())
reactor.listenTCP(1080, channel.HTTPFactory(site))
reactor.run()
I'm still getting a hang with files >= 10MB, using 0.2.0+svn20070403, but very glad to see this targetted for GoldMaster!
comment:10 Changed 9 years ago by therve
There is some stuff in the large-fileupload-1873 branch. The configuration of the limit is somewhat done. But the real bug mentioned here is not, because I don't manage to create an unit test for it.
comment:11 Changed 9 years ago by therve
comment:12 Changed 7 years ago by therve
- Owner therve deleted
comment:13 Changed 6 years ago by exarkun
- Resolution set to wontfix
- Status changed from new to closed
Not going to fix this, see #4821. Twisted Web doesn't have a similar problem with large uploads, so not retargeting at the web component either.
uploatserver.py