Ticket #1873 defect closed wontfix

Opened 7 years ago

Last modified 2 years ago

Large File Uploads Fail

Reported by: lennyg Owned by:
Priority: normal Milestone: Web2-Gold-Master
Component: web2 Keywords:
Cc: Branch:
Author: Launchpad Bug:

Description (last modified by jknight) (diff)

resource.PostableResource doesn't seem able to handle file uploads much larger than 9M. At 10M, the render() method is never entered. Here is a code snippet that can be used to reproduce the error: 

import os
from twisted.internet import reactor
from twisted.web2 import server, http, resource, channel
from twisted.web2 import http_headers, responsecode
from twisted.web2 import iweb, stream

SAVEDIR = "/tmp"
READSIZE=8192

class UploadFile(resource.PostableResource):
    def render(self, ctx):
        request = iweb.IRequest(ctx)
        filename = request.files['filename'][0][0]
        file = request.files['filename'][0][2]

        filestream = stream.FileStream(file)
        dest = os.path.join(SAVEDIR,filename)
        destfile = os.fdopen(os.open(dest,
                os.O_WRONLY | os.O_CREAT | os.O_EXCL,
                0644), 'w', 0)
        stream.readIntoFile(filestream, destfile)

        msg = "saved %s to %s" % (filename, dest)
        print msg
        return http.Response(stream="%s" % msg)

class Toplevel(resource.Resource):
    addSlash = True
    def render(self, ctx):
        return http.Response(responsecode.OK,
                {'content-type': http_headers.MimeType('text', 'html')},
                "Hello")

    child_uploadfile = UploadFile()

if __name__ == "__main__":
    site = server.Site(Toplevel())
    reactor.listenTCP(1080, channel.HTTPFactory(site))
    reactor.run()


And here is an html form that can be used to test, with files of varying sizes:

<html>
    <form action="http://localhost:1080/uploadfile"
            enctype="multipart/form-data" method="post">
        filename: <input type="file" name="filename">
        <input type="submit" value="submit">
    </form>
</html>

Programmatic clients (written in e.g. httplib) also get blocked upon writing only a few hunder KB if the file is larger than 10MB.

Attachments

uploadserver.py Download (1.1 KB) - added by lennyg 7 years ago.
uploatserver.py
uploadpost.html Download (209 bytes) - added by lennyg 7 years ago.
html form for upload to uploadserver.py

Change History

Changed 7 years ago by lennyg

uploatserver.py

Changed 7 years ago by lennyg

html form for upload to uploadserver.py

1

  Changed 7 years ago by jknight

  • description modified (diff)

Modified submission for proper formatting.

2

  Changed 7 years ago by lennyg

  • component changed from core to web2

Sorry -- didn't realize I could "Attach" (button didn't show up on entry screen) nor that inlined code would get all wonky.

3

  Changed 7 years ago by lennyg

James Y. Knight wrote ( http://twistedmatrix.com/pipermail/twisted-web/2006-June/002733.html):

The failing with > 10MB is intentional, but the hanging is not. It's supposed to return an error response to the client. In fileupload.py: 233, parseMultipartFormData has arguments which limit the number of fields and the size of the uploaded data. The default limit for size is 10MB.

Unfortunately, it looks like there is no way to pass in different limits to fileupload.parseMultipartFormData from where it's called in server.parsePOSTData.

So there's two bugs here: 1) hangs instead of returning a response. 2) limit is not configurable.

4

  Changed 7 years ago by jknight

Initial diagnostics show that it is indeed returning an error response but that the client doesn't get it. Perhaps because it's still blockingly trying to upload the rest of the file. It seems as though the server has paused reading the input (thus causing the client to block) rather than ignoring the rest.

5

  Changed 7 years ago by jknight

Yes, that is exactly it. The server is supposed to close the connection when responding to a request before the complete request has been read, which it currently doesn't do.

6

  Changed 7 years ago by dreid

  • owner changed from dreid to jknight

7

  Changed 7 years ago by glyph

  • milestone set to Web2-Gold-Master

8

follow-up: ↓ 9   Changed 6 years ago by therve

  • owner changed from jknight to therve

9

in reply to: ↑ 8   Changed 6 years ago by lennyg

Replying to therve:

the previously included code isn't right. the following demonstrates the problem properly: 

#!/usr/bin/python

import os
from twisted.internet import reactor
from twisted.web2 import server, http, resource, channel, stream
from twisted.web2 import http_headers, responsecode
from twisted.web2 import iweb

SAVEDIR = "/tmp"

class UploadFile(resource.PostableResource):
    def render(self, ctx):
        request = iweb.IRequest(ctx)
        filename = request.files['filename'][0][0]
        file = request.files['filename'][0][2]

        filestream = stream.FileStream(file)
        dest = os.path.join(SAVEDIR,filename)
        destfile = os.fdopen(os.open(dest,
            os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0644), 'w', 0)
        d = stream.readIntoFile(filestream, destfile)
        d.addBoth(self.cleanup, destfile, file,
                "saving %s to %s" % (filename, dest))
        return d

    def cleanup(self, r, destfile, file, msg):
        destfile.close()
        file.close()

        if r:
            print "error: %s" % r
            msg = "%s failed" % msg
        else:
            msg = "%s succeeded" % msg
        print msg
        return http.Response(stream="%s" % msg)

class Toplevel(resource.Resource):
    addSlash = True
    def render(self, ctx):
        return http.Response(responsecode.OK,
                {'content-type': http_headers.MimeType('text', 'html')},
                "Hello")

    child_uploadfile = UploadFile()

if __name__ == "__main__":
    site = server.Site(Toplevel())
    reactor.listenTCP(1080, channel.HTTPFactory(site))
    reactor.run()

I'm still getting a hang with files >= 10MB, using 0.2.0+svn20070403, but very glad to see this targetted for GoldMaster!

10

  Changed 6 years ago by therve

There is some stuff in the large-fileupload-1873 branch. The configuration of the limit is somewhat done. But the real bug mentioned here is not, because I don't manage to create an unit test for it.

11

  Changed 6 years ago by therve

(In [21310]) Merge forward work from large-fileupload-1873

Refs #2836 Refs #1873

12

  Changed 3 years ago by therve

  • owner therve deleted

13

  Changed 2 years ago by exarkun

  • status changed from new to closed
  • resolution set to wontfix

Not going to fix this, see #4821. Twisted Web doesn't have a similar problem with large uploads, so not retargeting at the web component either.

14

  Changed 2 years ago by <automation>

Note: See TracTickets for help on using tickets.