[Twisted-Python] Twisted-webserver hangs when serving big files

[Thomas Weholt]

I'm trying to create a webserver using code from HEP Message Server, which
in turn is based on Twisted. I cannot use rpys in this project due to
limitations on other parts of the system. I need to be able to send huge
files in a non-blocking manner, preferrably supporting Byte-ranges and
partial downloads too, but that's secondary to just being able to download
normally.

My code hangs when the request is for big files. Smaller ones like JPEGs
etc. is ok, but for a test-file on approx. 33MB it just hangs and starts
chewing up memory. The files are stored in a folder called wwwroot in the
same folder as the source file. I'm using Twisted 1.0.5.

The project is a normal webserver used for serving files from 10-500mb,
while also serving HTML-based content. I need to be able to control the
generated content, ie. a producer generates the core-content and I wrap that
content in a template which holds the overall design for the entire site. I
didn't find a way to do that using rpys so I'm trying this code instead. Ok,
that's not the important part, if somebody could just tell me why this
hangs. I can see that the server sends out the correct content-type, but it
seem to hang when it is suppose to write the binary content to the client.

PS! Any info on Twisted.webs status on Byte-ranges, partial download and
gzipped content in responses are also important.

Best regards,
Thomas Weholt

from twisted.protocols.http import HTTPFactory, HTTPChannel, Request
from twisted.internet import app
import time, os, sys, types, mimetypes, ConfigParser

def getFactory(config):
    f = HTTPFactory()
    f.config = config
    f.protocol = MyChannel
    return f

class MyChannel(HTTPChannel):
    def __init__(self):
        HTTPChannel.__init__(self)
        self.requestFactory = RequestHandler

class RequestHandler(Request):
    def __init__(*args):
        Request.__init__(*args)
        self = args[0]

    def process(self):

        if not self.authenticate():
            self.setResponseCode(401, "Unauthorized")
            self.setHeader('WWW-Authenticate', 'Basic realm="Some server"')
            self.write("Access denied")
            self.finish()
            return

        requestData = self.uri.split("?")
        self.pageName = requestData[0]
        if len(requestData) > 1:
            form = parse_qs(requestData[1])
        else:
            form = {}

        self.pageName = 'somehugefile.dat'
        self.sendRequestedFile()
        self.finish()

    def getLocalFileName(self, pageName):
        filenameParts = ['wwwroot'] + pageName.split("/")
        pagefilename = os.path.join(*filenameParts)
        if os.path.isfile(pagefilename):
            return pagefilename

    def sendRequestedFile(self):
        pagefilename = self.getLocalFileName(self.pageName)
        mimeInfo = mimetypes.guess_type(pagefilename)
        if mimeInfo[0]:
            mimetype = mimeInfo[0]
        else:
            mimetype = "text/plain"
        htmlFile = open(pagefilename, 'r+b')
        self.setHeader("Content-type",mimetype)
        chunk = htmlFile.read(1024)
        while chunk:
            self.write(chunk)
            chunk = htmlFile.read(1024)
        htmlFile.close()

    def authenticate(self):
        username = self.getUser()
        password = self.getPassword()
        if username and password:
            return username == 'thomas' and password == 'test.pass'

print "Running something ...",
app = app.Application("MyApp")
config = ConfigParser.ConfigParser()
config.readfp(open('config.ini'))
app.listenTCP(6060, getFactory(config))
print "ok"
app.run()
print "Stopped"