[Twisted-Python] Strange recursion error with twisted.web

[Peter Westlake]

On Tue, 9 Mar 2021, at 19:28, Glyph wrote:
> 
> 
> > On Mar 9, 2021, at 4:54 AM, Peter Westlake <peter.westlake at pobox.com> wrote:
> > 
> > I'm getting a "maximum recursion depth exceeded" error that appears to be coming from flatten(). The odd thing is that it only happens sometimes. The HTML that's being flattened does have a few Deferreds in it. Those come from function calls, which cache the results, which might explain why I only see the error on the first visit to the page (as far as I can tell). 
> > 
> > The system recursion limit is the standard 1000. My HTML is only nested a few tags deep, two orders of magnitude short of that. Is there anything about the way flatten() works that might cause this behaviour?
> 
> flatten() can definitely result in some deep recursive stacks, 
> particularly in combination with synchronous Deferreds which have their 
> own accumulating stack costs. I'd be interested to see a minimal 
> reproducer for this though, I'm sure we could do a lot better.

Here it is:

import sys
from twisted.internet import reactor, defer, task
from twisted.web.template import flatten

def output(stuff):
    sys.stdout.write(stuff.decode())

def sync(reactor):
    return flatten(None, [defer.succeed(str(i)+'\n') for i in range(1000)], output)

task.react(sync)


It fails after printing 197 lines. The same sort of thing using deferLater instead of defer.succeed printed 1000 without error.

Peter.