Unbounded recursion in twisted/web2/stream.py

[Jean-Paul Calderone]

StreamProducer.resumeProducing and StreamProducer._doWrite can mutually recurse unboundedly. Here's part of a traceback which occurred when they did:

2006/08/07 10:15 CDT [HTTPChannel,12147,198.49.126.190] Unhandled Error
        Traceback (most recent call last):
          File "/home/trac/Projects/Twisted/trunk/twisted/python/log.py", line 53, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "/home/trac/Projects/Twisted/trunk/twisted/python/log.py", line 38, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "/home/trac/Projects/Twisted/trunk/twisted/python/context.py", line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/home/trac/Projects/Twisted/trunk/twisted/python/context.py", line 37, in callWithContext
            return func(*args,**kw)
        --- <exception caught here> ---
          File "/home/trac/Projects/Twisted/trunk/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite
            why = getattr(selectable, method)()
          File "/home/trac/Projects/Twisted/trunk/twisted/internet/abstract.py", line 127, in doWrite
            self.producer.resumeProducing()
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/stream.py", line 744, in resumeProducing
            self._doWrite(data)
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/stream.py", line 765, in _doWrite
            self.resumeProducing()
...
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/stream.py", line 744, in resumeProducing
            self._doWrite(data)
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/stream.py", line 762, in _doWrite
            self.consumer.write(data)
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/channel/http.py", line 447, in write
            self.transport.write(data)
          File "/home/trac/Projects/Twisted/trunk/twisted/internet/abstract.py", line 175, in write
            self.startWriting()
        exceptions.RuntimeError: maximum recursion depth exceeded

[camrdale]

I just hit this bug as well. The original poster didn't mention it, but the clipped Traceback showed (for me anyway) 484 calls to resumeProducing, and 483 calls to _doWrite.

I think it only occurs in unusual situations where the stream produces small amounts of bytes at a time. In my case, my stream is a modified FileStream that will read a max of 4096 bytes at a time, whereas the default FileStream would read much more using mmap. The number of times the stream.read() gets called increases the chance of hitting this bug.

The obvious and easy fix is to replace one of the calls with a reactor.callLater(0, ...), either at line 744 or 765 of stream.py. Probably the call to resumeProducing in _doWrite at line 765 should be replaced with reactor.callLater(0, self.resumeProducing).

[hanser]

Replying to camrdale:

I just hit this bug as well. The original poster didn't mention it, but the clipped Traceback showed (for me anyway) 484 calls to resumeProducing, and 483 calls to _doWrite.

I think it only occurs in unusual situations where the stream produces small amounts of bytes at a time. In my case, my stream is a modified FileStream that will read a max of 4096 bytes at a time, whereas the default FileStream would read much more using mmap. The number of times the stream.read() gets called increases the chance of hitting this bug.

The obvious and easy fix is to replace one of the calls with a reactor.callLater(0, ...), either at line 744 or 765 of stream.py. Probably the call to resumeProducing in _doWrite at line 765 should be replaced with reactor.callLater(0, self.resumeProducing).

May be the case does not occur so often But for me,

• it occurred at my first attempt at using web2, when i tried displaying an array

with 300 lines.

• I discovered it is fixed in the twisted version from cpushare

[Jean-Paul Calderone]

Not going to fix this, see #4821. Twisted Web has no similar defect as far as I know, so not retargeting this ticket a the web component.