[Twisted-Python] Deferreds vs sys.getrecursionlimit()

[Jean-Paul Calderone]

On Mon, 17 Nov 2008 15:49:41 -0800, Brian Warner <warner at lothar.com> wrote:
>> Ideally, Deferred just shouldn't have this problem. If we can't eliminate
>> the problem entirely, then we can at least add a more useful error message
>> which can explain how you can start debugging.
>
>Yeah, when I last looked into this (a couple years ago), I figured that the
>Deferred doesn't have enough information to safely optimize out the
>tail-call. You never know who else might have a handle on the Deferred and
>might add a new callback to it. It once occurred to me that it might be
>easier to do this safely if Deferred were broken up into two pieces (like E's
>Promise/Resolver pair: basically one side would get .callback and .errback,
>while the other side would get .addCallbacks/etc), but I didn't pursue that
>thought very far.

Of course, the best response to this would be an implementation of the
iterative version of _runCallbacks.  However, I do think it is possible
to get rid of this recursion.  It doesn't really matter who else might
have a reference to either Deferred involved.  The new frame going onto
the stack is just another Deferred._runCallbacks (unless a subclass
overrides it, but DeferredList is the only subclass in Twisted, and we
should really deprecated it, and continue to discourage people from
subclassing Deferred, and _runCallbacks is private anyway so there).
The recurser (ie, the Deferred._runCallbacks doing the `self.result.addBoth(
self._continue)´ knows how the recursee (ie, the Deferred having addBoth
called on it) behaves - just like itself.  The obvious transformation
(inlining a bunch of code from outside of _runCallbacks into _runCallbacks)
will result in something that's really ugly, but it should work.  And I
think there is probably an approach that's less ugly, too.

This addresses only one of the problems you raised, but it's the one I
think Glyph was talking about eliminating by changing the implementation
of Deferred.

It's possible there's a way to remove the other one with an implementation
change to Deferred as well (but it's not as clear to me what that change is
yet).  However, it's much easier to avoid that one by writing code in a
slightly different way.

eventual-send is one different way, but there are also other more efficient
approaches which are also always correct.  These generally take the form
of avoiding creating a giant stack of Deferreds in the first place by only
changing each Deferred which would have been "interior" on that stack to
the one immediately beneath it and chaining the bottom directly to the top.

Jean-Paul