[Twisted-Python] Unjelly - recursion limit reached

[David K. Hess]

On Nov 11, 2005, at 8:46 AM, Jean-Paul Calderone wrote:

> On Fri, 11 Nov 2005 08:26:03 -0600, "David K. Hess"  
> <dhess at verscend.com> wrote:
>>
>> On Nov 7, 2005, at 8:42 AM, David K. Hess wrote:
>>> Bottom line, if all of these things are true, then I'm not sure   
>>> what is going wrong. Is it possible that jelly is at times  
>>> walking  my object graph (which is interconnected to a certain  
>>> degree and  changes dynamically) in such a way that it has to  
>>> perform a  significant amount of recursion? In other words, maybe  
>>> this isn't a  bug but I just need to increase the runtime  
>>> recursion limit?
>>>
>>> FYI, here's a snippet of the exception:
>>
>> I hate replying to my own post but I'm beginning to believe that   
>> recursion is the root of the problem and want to focus on it.  
>> From  the exception trace, it looks like it takes on average 3.5  
>> stack  frames to unjelly a nested object. With a default recursion  
>> limit of  1000 stack frames, this means that Twisted is limited to  
>> transferring  an object graph that has a depth first traversal  
>> through it of length  285 objects (roughly speaking). You could of  
>> course easily create a  linked list of 285 objects to force the  
>> situation.
>
> Linked list?  Hee hee.  Crazy.
>
>>
>> This is a pretty scary limitation. Is anybody working on a non-  
>> recursion based version of jelly? Am I going to have to move to   
>> Stackless Python?
>
> You can raise the recursion limit.  See sys.setrecursionlimit().

Yes, but with the standard Python build on Windows, you can only go  
up to 2000. So now, I can do 570 objects. I'm working on trying to  
track down editbin.exe so I can raise the stack size on python.exe  
and see how much farther I can go.

> Stackless has nothing to do with this.  It won't help fix this  
> problem.

Are we talking about the same thing? The key architectural feature of  
Stackless Python is "practically unlimited recursion". Forget about  
all the tasklet and channel stuff.

> The real way to "fix" this is to re-implement the jellying process  
> iteratively.  Or, don't use such deeply nested objects.

My object graph is a natural and clean data structure. I don't want  
to lose that just because of limitations in my tools.

Is this currently being addressed or in a road map? Am I unknowingly  
volunteering for it? :-o

Dave

>
> Jean-Paul
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python