[Twisted-web] Exceptions in twisted.web.client

Chris Were chris.were at gmail.com
Tue Oct 6 23:33:11 EDT 2009 

Hi,
I have an application that fetches many urls, following any redirects. Every
so often I get a variety of unhandled exceptions from within
twisted.web.client (see snippet at the bottom of email).

My code has an addErrback attached to the deferred and a try/except around
the getPage() call, neither of which get invoked.

Are these exceptions that web.client needs to catch internally and then call
self.deferred.errback so that my error handler will receive it?

If not, how can I catch them?

Cheers,
Chris

--- Exception 1 ---

2009-10-06 20:21:16-0700 [HTTPPageGetter,client] Unhandled Error
 Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py",
line 84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py",
line 69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py",
line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py",
line 37, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/selectreactor.py",
line 146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py",
line 463, in doRead
    return self.protocol.dataReceived(data)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py",
line 231, in dataReceived
    why = self.lineReceived(line)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/http.py",
line 420, in lineReceived
    self.handleEndHeaders()
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py",
line 74, in handleEndHeaders
    m()
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py",
line 93, in handleStatus_301
    _parse(url, defaultPort=self.transport.getPeer().port)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py",
line 426, in _parse
    port = int(port)
exceptions.ValueError: invalid literal for int() with base 10: ''

--- Exception 2 ---

2009-10-06 20:28:42-0700 [HTTPPageGetter,client] Unhandled Error
Traceback (most recent call last):
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py",
line 84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/log.py",
line 69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py",
line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py",
line 37, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/selectreactor.py",
line 146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/tcp.py",
line 463, in doRead
    return self.protocol.dataReceived(data)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/protocols/basic.py",
line 231, in dataReceived
    why = self.lineReceived(line)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/http.py",
line 420, in lineReceived
    self.handleEndHeaders()
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py",
line 74, in handleEndHeaders
    m()
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py",
line 93, in handleStatus_301
    _parse(url, defaultPort=self.transport.getPeer().port)
  File
"/usr/local/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/web/client.py",
line 425, in _parse
    host, port = host.split(':')
exceptions.ValueError: too many values to unpack

--- Exception 3 ---

2009-10-06 20:31:11-0700 [HTTPPageGetter,client] Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", line
84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", line
69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/local/lib/python2.6/site-packages/twisted/python/context.py",
line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python2.6/site-packages/twisted/python/context.py",
line 37, in callWithContext
    return func(*args,**kw)
 --- <exception caught here> ---
  File
"/usr/local/lib/python2.6/site-packages/twisted/internet/selectreactor.py",
line 146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "/usr/local/lib/python2.6/site-packages/twisted/internet/tcp.py",
line 463, in doRead
    return self.protocol.dataReceived(data)
  File "/usr/local/lib/python2.6/site-packages/twisted/protocols/basic.py",
line 231, in dataReceived
    why = self.lineReceived(line)
  File "/usr/local/lib/python2.6/site-packages/twisted/web/http.py", line
404, in lineReceived
    status = l[1]
exceptions.IndexError: list index out of range
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-web/attachments/20091007/310aeb7e/attachment.htm

More information about the Twisted-web mailing list