[Twisted-Python] The Real Error (tm) [was Re: how to write a safe catch-all]

[Chris Withers]

On 30/09/2010 15:23, Phil Mayers wrote:
>> def loop():
>> d = maybeDeferred(doStuff)
>> d.addErrback(partial(log.err,_why='Unhandled scheduled exception'))
>
>> So, how come my log.err doesn't get used for the AttributeError on
>> connector?
>
> If you mean in your most recent "test_looping.py" example, it still uses
> reactor.callLater. The call stack has gone away by the time the call is
> made, so the exception just propagates up to the top level, where it's
> logged.

So, I appear to be back to the case where I can either gracefully handle 
the exception *or* gracefully handle the errback, but not both?

> I honestly think a more complete example showing the real Twisted API
> that's causing you unhandled errors would help here.

Unfortunately, you neither want to see, nor am I allowed to publish to a 
mailing list, that several hundred lines of proprietary code I'd need to 
post...

As far as the original error goes, we hit the problem using 
twisted.protocols.ftp.FTPClient to ftp a file up to a remote host.
That remote host only accepts active ftp transfers. As a result of 
either of both of our firewall or FTPClient not handling active ftp and 
only handling passive ftp, we end up seeing the following logging as the 
scheduler dies:

2010-09-27 15:30:16,340 ERROR   : log         (24331|7f2e47b4d6e0):
Unhandled exception sending schedule transmission
Traceback (most recent call last):
    File
"Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/python/context.py", line
37, in callWithContext
      return func(*args,**kw)
    File
"Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/selectreactor.py", line 

146, in _doReadOrWrite
      why = getattr(selectable, method)()
    File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py",
line 631, in doConnect
      self.failIfNotConnected(error.getConnectError((err, strerror(err))))
    File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py",
line 588, in failIfNotConnected
      del self.connector
--- <exception caught here> ---
    File "ourcode.py", line 180, in checkSchedule
      yield self.sendTransmissions(...)
exceptions.GeneratorExit:

2010-09-27 15:30:28,428 ERROR   : log         (24331|7f2e47b4d6e0):
Unhandled error in Deferred:
2010-09-27 15:30:28,584 ERROR   : log         (24331|7f2e47b4d6e0):
Unhandled Error
Traceback (most recent call last):
Failure: twisted.protocols.ftp.FTPError: ('Connection Failed',
<twisted.python.failure.Failure <class
'twisted.internet.error.ConnectError'>>)

I can't quite make sense of the above, which is why I distilled it down 
to as-small-as-possible a piece of code that shows the type of 
exceptions and errbacks I need to deal with...

cheers,

Chris


-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk