[Twisted-Python] ludicrous timeouts in protocols.http.HTTPFactory and web.server.Site

Antoine Pitrou solipsis at pitrou.net
Fri Dec 31 04:50:44 EST 2004


> However, for very large POST requests over a slow link (ever tried to attach
> a large file to your webmail over dial-up?), 30 or 60 seconds is still too
> slow, but I could see an argument for, say, 30 minutes being the default.

Well, yes, actually it depends on which timeout we are really testing.
The timeout I had problems with is when a request has already been
answered, but the client leaves its socket open for whatever reason,
without sending or waiting for anything: so it's a case of "keepalive"
HTTP connection that is kept alive much too long, I think. I agree that
if data is being transfered in whatever direction (whether request
payload or reply payload), the connection should not be closed ;)

Since this happened with a well-known proxy server (according to the Via
header, it is "NetCache NetApp/5.6.1"), some other people might
encounter the same problem and wonder why their low traffic HTTP server
sometimes runs out of file descriptors after 30 minutes of activity. I
must insist that this happened with only _one_ client sending XMLRPC
requests every 2 or 3 seconds.


By the way, I've written a little routine to detect HTTP proxy settings.
It works under Windows and Linux, but could surely be improved. Here it
is in case it interests someone:

import urlparse

def discover_proxy():
    """
    Returns a (host, port) tuple if a proxy is found in the
    current machine configuration, (None, None) otherwise.
    """

    host_port = None

    # Un*x et al.
    if 'http_proxy' in os.environ:
        parts = urlparse.urlparse(os.environ['http_proxy'])
        if not parts[0] or parts[0] == 'http':
            host_port = parts[1]

    # Windows
    try:
        import _winreg as winreg
    except ImportError:
        pass
    else:
        try:
            # Try to grab current proxy settings from the registry
            regkey = winreg.OpenKey(winreg.HKEY_CURRENT_USER,
                'Software\\Microsoft\\Windows\\CurrentVersion\\Internet Settings')
            regval = winreg.QueryValueEx(regkey, 'ProxyServer')
            regkey.Close()
            regval = str(regval[0])
            # Regval can be of two types:
            # - 'myproxy:3128' if one proxy for all protocols
            # - 'ftp=myftpproxy:3128;http=myhttpproxy:3128;...' if several different proxies
            values = regval.split(';')
            if len(values) > 1:
                for s in values:
                    scheme, p = s.split('=')
                    if scheme == 'http':
                        host_port = p
                        break
            else:
                host_port = values[0]

        except Exception, e:
            print str(e)
            pass

    # Split host and port
    if host_port is not None:
        t = host_port.split(':')
        host = t[0].strip()
        if host:
            try:
                port = int(t[1])
            except:
                port = 80
            return host, port

    return None, None


Regards

Antoine.






More information about the Twisted-Python mailing list