[Twisted-Python] Deferreds vs sys.getrecursionlimit()

Brian Warner warner at lothar.com
Mon Nov 17 16:49:41 MST 2008


> Ideally, Deferred just shouldn't have this problem. If we can't eliminate
> the problem entirely, then we can at least add a more useful error message
> which can explain how you can start debugging.

Yeah, when I last looked into this (a couple years ago), I figured that the
Deferred doesn't have enough information to safely optimize out the
tail-call. You never know who else might have a handle on the Deferred and
might add a new callback to it. It once occurred to me that it might be
easier to do this safely if Deferred were broken up into two pieces (like E's
Promise/Resolver pair: basically one side would get .callback and .errback,
while the other side would get .addCallbacks/etc), but I didn't pursue that
thought very far.

Using an eventual-send is unfortunate but correct (in that it will reliably
avoid the problem, but it's probably a noticable performance hit to blow away
the entire stack for each call). An unwise-but-less-unfortunate approach
would be to use an eventual-send only when it appears necessary, as in the
following strawman:

class Deferred:
    def _continue(self, result):
        self.result = result
        if len(traceback.extract_stack()) > sys.getrecursionlimit() - 100:
            eventually(self.unpause)
        else:
            self.unpause()


It might be enough to have Defer._runCallbacks() look to see if the result of
callback() is a recursion-depth-exceeded RuntimeError and do something
special in response to it. Zooko showed me some code that would temporarily
raise sys.setrecursionlimit() so that the error could be Failure-ized
properly.. maybe that would be enough. A lot of the frustration cause by this
sort of problem is that the Failure-rendering code runs out of stack space
too.

cheers,
 -Brian




More information about the Twisted-Python mailing list