[Twisted-Python] Problem exiting script that calls a func returning a called deferred

[Terry Jones]

>From searching and reading the list archives, I know the following comes up
under various guises, and although I think I understand what's going on, I
don't see a nice solution.

I'm trying to write a standalone script. It calls externalFunc, a function
someone else wrote that returns a deferred. My script could look something
like this:

    from twisted.internet import reactor
    from twisted.python import log

    if __name__ == '__main__':
        def ok(result):
            print "ok called with", result
        def nok(failure):
            print "nok called with", failure
            return failure
        def stop(x):
            reactor.stop()

        d = externalFunc()
        d.addCallback(ok)
        d.addErrback(nok)
        d.addBoth(stop)
        d.addErrback(log.err)
        reactor.run()

And that works fine.

Now suppose I want to do some testing in which I swap out the external
function for a library I wrote myself (or for another implementation).  In
the testing library, externalFunc is simplified and uses defer.success() to
return its result.

In this case the script raises RuntimeError, "can't stop reactor that isn't
running" because the deferred that comes back from externalFunc has already
been called. So when I add the call/errbacks, they are fired right away and
the reactor isn't running when the stop function tries to stop it.

And script never exits because it then starts the reactor.

At this point there are various contortions I can try. Catching the
RuntimeError works, but the script never exits. Catching the RuntimeError
and then calling sys.exit doesn't work because sys.exit raises SystemExit
and that is caught in twisted/internet/defer.py _runCallbacks.

Supposing I had more control, the calling function could create the
deferred, pause it, pass it to externalFunc, add callbacks, then unpause.
But that doesn't work either as the callbacks fire as soon as unpause is
called, and you get the reactor not started error, followed by no exit as
you then start the reactor.

Testing reactor.running before trying to stop it in stop gets rid of the
error, but the reactor is then started and not stopped.

The only bulletproof way to deal with this that I've come up with is to
call self.fireSystemEvent("shutdown") but that feels wrong.

Several solutions seem possible with changes to Twisted: don't catch
SystemExit; give reactor.stop a (default False) keyword arg to tell it to
ignore a stopped reactor; stop deferreds from firing call/err backs if they
are paused (IMO it's a bit non-paradigmatic to have call/errback events
firing when the reactor isn't even running), etc.

I suppose I'm just not doing things right. Can someone tell me how I should
be doing this?

Terry