[Twisted-Python] how to write a safe catch-all

[Chris Withers]

On 28/09/2010 15:21, exarkun at twistedmatrix.com wrote:
>> But, more crucially, the looping call then appears to stop.
>
> The function you're looping over returns a Deferred that never fires.
> The LoopingCall isn't stopped, it's waiting for the Deferred.

So, inlineCallbacks/generator things will only process errbacks, not 
actual exceptions raised inside asyncronous code called from them?!

>> What can I do to get the exception logged and then everything handled
>> sanely such that the looping call can continue and my lopp function
>> will keep getting called once every second rather than stopping?
>
> When you do reactor.callLater(n, f), you put f into an execution context
> where the only thing that will happen to exceptions it raises is that
> they will be logged.

Okay, the script was a simplification of the "real problem" to try and 
give the list a "smallest failing example to try and help with".

The real logging looks like this:

2010-09-27 15:30:16,340 ERROR   : log         (24331|7f2e47b4d6e0): 
Unhandled exception sending schedule transmission
Traceback (most recent call last):
   File 
"Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/python/context.py", line 
37, in callWithContext
     return func(*args,**kw)
   File 
"Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/selectreactor.py", line 
146, in _doReadOrWrite
     why = getattr(selectable, method)()
   File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py", 
line 631, in doConnect
     self.failIfNotConnected(error.getConnectError((err, strerror(err))))
   File "Twisted-8.2.0-py2.5-linux-x86_64.egg/twisted/internet/tcp.py", 
line 588, in failIfNotConnected
     del self.connector
--- <exception caught here> ---
   File "ourcode.py", line 180, in checkSchedule
     yield self.sendTransmissions(...)
exceptions.GeneratorExit:

2010-09-27 15:30:28,428 ERROR   : log         (24331|7f2e47b4d6e0): 
Unhandled error in Deferred:
2010-09-27 15:30:28,584 ERROR   : log         (24331|7f2e47b4d6e0): 
Unhandled Error
Traceback (most recent call last):
Failure: twisted.protocols.ftp.FTPError: ('Connection Failed', 
<twisted.python.failure.Failure <class 
'twisted.internet.error.ConnectError'>>)

I don't quite follow what the above is trying to tell me, other than an 
FTP connection failed. However, I don't understand why that results in a 
GeneratorExit rather than an errback of the original exception being 
caught by the top level handler in the loop() function (switching back 
to the example terminology for simplicity). I also don't understand why 
an unhandled deferred is being logged rather than fed back into the 
handler I built for it!

> exception, then you have to arrange for that.  You can do this by
> wrapping f with another function that handles the exception and sends it
> where you want.

Well, as far as I can tell, that's what I'm trying to do. However, the 
thing failing in the real world is in code I don't "own" (looks like 
twisted's inards...) and I'd like to be able to cater for any failure, 
unforseen or not (barring say a SyntaxError ;-)) and still have the 
loop() call keep doing its thing.

How can I do that?

Chris