[Twisted-web] Re: web client & "TimeoutError: User timeout caused connection failure"

[Lenny G Arbage]

--- Lenny G Arbage <alengarbage at yahoo.com> wrote:
>
> (I can produce a short example that reproduces the
> behavior if needed)

'ere it is:

import sys
from twisted.python import failure
from twisted.internet import reactor, defer
from twisted.web.client import getPage

CONCURRENT=800

def itersuccess(res, message):
    print "itersuccess: %s" % message
    return res

def itererror(failure, message):
    print "itererror message: %s" % message
    failure.printTraceback()
    return failure

def testGetPage(url, num=CONCURRENT):
    print "testGetPage started..."
    dlist = []
    for i in range(num):
        deferred = getPage(url)
        deferred.addCallback(itersuccess,
                "succeeded at testGetPage %d" % i)
        deferred.addErrback(itererror,
                "failed at testGetPage %d" % i)
        dlist.append(deferred)
    return defer.DeferredList(dlist,
consumeErrors=True)

def runTests(url):
    d1 = testGetPage(url)
    d1.addBoth(cleanup)
    reactor.run()

def cleanup(res):
    reactor.stop()

if __name__ == '__main__':
    runTests(sys.argv[1])


I can run this against a local version of apache, and
get a bunch of successes until somewhere in the 600s:

itersuccess: succeeded at testGetPage 679
itersuccess: succeeded at testGetPage 657
itersuccess: succeeded at testGetPage 683
itererror message: failed at testGetPage 638
Failure: twisted.internet.error.TimeoutError: User
timeout caused connection failure.
itererror message: failed at testGetPage 639
Failure: twisted.internet.error.TimeoutError: User
timeout caused connection failure.

Your results, of course, will vary depending on
hardware and software.  With other servers I only need
to get in 50-60s before I start getting the
TimeoutError messages.

Is the answer to just catch these messages and keep
retrying (after a short pause)?

Lenny

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com