[Twisted-Python] Unjelly - recursion limit reached
David K. Hess
dhess at verscend.com
Fri Nov 11 10:11:42 EST 2005
On Nov 11, 2005, at 8:46 AM, Jean-Paul Calderone wrote:
> On Fri, 11 Nov 2005 08:26:03 -0600, "David K. Hess"
> <dhess at verscend.com> wrote:
>>
>> On Nov 7, 2005, at 8:42 AM, David K. Hess wrote:
>>> Bottom line, if all of these things are true, then I'm not sure
>>> what is going wrong. Is it possible that jelly is at times
>>> walking my object graph (which is interconnected to a certain
>>> degree and changes dynamically) in such a way that it has to
>>> perform a significant amount of recursion? In other words, maybe
>>> this isn't a bug but I just need to increase the runtime
>>> recursion limit?
>>>
>>> FYI, here's a snippet of the exception:
>>
>> I hate replying to my own post but I'm beginning to believe that
>> recursion is the root of the problem and want to focus on it.
>> From the exception trace, it looks like it takes on average 3.5
>> stack frames to unjelly a nested object. With a default recursion
>> limit of 1000 stack frames, this means that Twisted is limited to
>> transferring an object graph that has a depth first traversal
>> through it of length 285 objects (roughly speaking). You could of
>> course easily create a linked list of 285 objects to force the
>> situation.
>
> Linked list? Hee hee. Crazy.
>
>>
>> This is a pretty scary limitation. Is anybody working on a non-
>> recursion based version of jelly? Am I going to have to move to
>> Stackless Python?
>
> You can raise the recursion limit. See sys.setrecursionlimit().
Yes, but with the standard Python build on Windows, you can only go
up to 2000. So now, I can do 570 objects. I'm working on trying to
track down editbin.exe so I can raise the stack size on python.exe
and see how much farther I can go.
> Stackless has nothing to do with this. It won't help fix this
> problem.
Are we talking about the same thing? The key architectural feature of
Stackless Python is "practically unlimited recursion". Forget about
all the tasklet and channel stuff.
> The real way to "fix" this is to re-implement the jellying process
> iteratively. Or, don't use such deeply nested objects.
My object graph is a natural and clean data structure. I don't want
to lose that just because of limitations in my tools.
Is this currently being addressed or in a road map? Am I unknowingly
volunteering for it? :-o
Dave
>
> Jean-Paul
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
More information about the Twisted-Python
mailing list