[Twisted-Python] Maximum recursion depth reached
sflist at ihonk.com
Fri May 9 14:30:41 EDT 2008
On Fri, 2008-05-09 at 13:40 -0400, Jean-Paul Calderone wrote:
> On Fri, 09 May 2008 10:18:57 -0700, Steve Freitas <sflist at ihonk.com>
> >I guess I'm a little mystified by what's going wrong here -- I chain
> >deferreds all the time -- I'm not sure what it is about returning a few
> >thousand deferreds wrapped up into a smaller number of DeferredLists
> >that's causing the problem, and I'd like to understand why.
> Sitting at the core of Deferred is a loop over callback functions. If
> you have a chained Deferred, then one of those callbacks is doing to be
> a function which recursively calls the function which has that loop. If
> you chain enough Deferreds, then eventually this recursion will fail. It
> turns out to be about 250 Deferreds (roughly 4 stack frames per level of
> chaining, with the default limit of 1000 stack frames imposed by CPython)
> which will trigger this limit.
Yep, that helps, Jean-Paul, thanks for the info. I've been chaining
deferreds for years, but I never thought there was a limit to how many
times I could do it, and even if I had, I'd have never expected the
number would be so low. I looked in the docs and found no mention of it.
> It may be possible to replace this recursion with iteration, but I'm not
> sure that solves all problems. After all, if you suddenly have to blow
> through thousands of levels of chaining in response to one event, then
> you're paying a pretty hefty price which could be avoided by jumping over
> all the irrelevant intermediate stuff. Maybe *that* could be implemented
> in Deferred as well somehow, but it's not totally obvious to me how. :)
I disagree that this would be a problem. If I'm chaining 10,000
deferreds, I'm taking responsibility to handle a blow-up of that
magnitude. That's why my original example involved putting 5 deferreds
at a time into a DeferredList -- in my real code I'm checking the
results from that DeferredList and stopping sanely if there's an issue.
Deferred's got too much mojo for me to consider submitting a patch, but
I hope somebody does, since there's no conceptual reason why there
should be such a limit.
/me wanders off to Trac to submit a ticket...
More information about the Twisted-Python