[Twisted-Python] Maximum recursion depth reached

Jean-Paul Calderone exarkun at divmod.com
Fri May 9 13:40:26 EDT 2008


On Fri, 09 May 2008 10:18:57 -0700, Steve Freitas <sflist at ihonk.com> wrote:
>On Fri, 2008-05-09 at 13:20 +0200, Maarten ter Huurne wrote:
>> By returning the DeferredList from next_batch(), you are chaining Deferreds:
>>
>> http://twistedmatrix.com/projects/core/documentation/howto/defer.html#auto11
>>
>> If you remove the "return" from next_batch(), the problem disappears. If you
>> want to be able to register a callback when all items are processed, it's
>> probably better to create a dedicated Deferred for that instead of chaining
>> the DeferredLists.
>
>Thanks for your reply, Marteen!
>
>I guess I'm a little mystified by what's going wrong here -- I chain
>deferreds all the time -- I'm not sure what it is about returning a few
>thousand deferreds wrapped up into a smaller number of DeferredLists
>that's causing the problem, and I'd like to understand why.

Sitting at the core of Deferred is a loop over callback functions.  If
you have a chained Deferred, then one of those callbacks is doing to be
a function which recursively calls the function which has that loop.  If
you chain enough Deferreds, then eventually this recursion will fail.  It
turns out to be about 250 Deferreds (roughly 4 stack frames per level of
chaining, with the default limit of 1000 stack frames imposed by CPython)
which will trigger this limit.

It may be possible to replace this recursion with iteration, but I'm not
sure that solves all problems.  After all, if you suddenly have to blow
through thousands of levels of chaining in response to one event, then
you're paying a pretty hefty price which could be avoided by jumping over
all the irrelevant intermediate stuff.  Maybe *that* could be implemented
in Deferred as well somehow, but it's not totally obvious to me how. :)

Hope this helps,

Jean-Paul




More information about the Twisted-Python mailing list