[Twisted-Python] how to write a safe catch-all

Phil Mayers p.mayers at imperial.ac.uk
Thu Sep 30 08:44:42 EDT 2010

On 30/09/10 11:48, Chris Withers wrote:
> On 30/09/2010 04:45, exarkun at twistedmatrix.com wrote:
>> On 12:52 am, chris at simplistix.co.uk wrote:
>>> Because I haven't found any permutation that doesn't result in the
>>> LoopingCall's calls to `loop` from stopping after the exception.
>>> I would be more than ecstatic to be proved wrong ;-)
>> You keep saying the LoopingCall stops.  It does not stop.  It is waiting
>> for a result.
> What result is it waiting for and how do I pass that from the `loop`
> function in my first example when the exception is caught?

It is waiting for the deferred you return to callback/errback.

This, I found, was initially non-obvious. I assumed that LoopingCall 
would be called every N seconds regardless, but it won't. If you return 
a deferred, the LoopingCall will stop until the deferred 
callback/errback's, then it will be restarted.

This is very useful in some circumstances.

>> If you provide it with a result, it will proceed.
> See above: how do I do that?
>> Glyph
>> suggested a number of (hackfully) mechanisms you might use to provide it
>> with a result.
> Why do you say hackfully? I'm trying to build a scheduler that is

Because this is (apparently) a bug in the underlying Twisted code, so 
anything you do to fix that is a hack.

The quickest way I can describe it is this; as I understand, any Twisted 
function which returns a deferred should never raise an Exception. The 
exception should instead flow down the errback.

This is because (as you've discovered) there are circumstances when a 
function is called in Twisted where the exception can't propagate "up" 
to the caller, because the caller is no longer higher in the call stack. 
Most prominent is reactor.callLater.

> resilient to failure in a scheduled task. Yes, of course I want to fix
> that failure, but I don't want a failure in one scheduled task to
> prevent any further scheduled tasks from being executed...
>> I suggested another way (fix the broken code, or at
>> least supply enough information for someone else to fix it).
> For all I know, it's already fixed in a newer version of Twisted
> (unfortunately, I can't move this project to a newer version of Twisted)
> but once I have a robust scheduler, I'll certainly do my best to report
> the underlying failure to you guys properly...

If you can produce a short example that uses the real Twisted API which 
fails, people might a) be able to find/fix the bug and b) suggest a less 
hacky way of ensuring you see the error.

More information about the Twisted-Python mailing list