Opened 16 years ago

Closed 11 years ago

#1993 defect closed wontfix (wontfix)

Unbounded recursion in twisted/web2/stream.py

Reported by: Jean-Paul Calderone Owned by:
Priority: normal Milestone: Web2-Gold-Master
Component: web2 Keywords:
Cc: Branch:
Author:

Description (last modified by Jean-Paul Calderone)

StreamProducer.resumeProducing and StreamProducer._doWrite can mutually recurse unboundedly. Here's part of a traceback which occurred when they did:

2006/08/07 10:15 CDT [HTTPChannel,12147,198.49.126.190] Unhandled Error
        Traceback (most recent call last):
          File "/home/trac/Projects/Twisted/trunk/twisted/python/log.py", line 53, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "/home/trac/Projects/Twisted/trunk/twisted/python/log.py", line 38, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "/home/trac/Projects/Twisted/trunk/twisted/python/context.py", line 59, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/home/trac/Projects/Twisted/trunk/twisted/python/context.py", line 37, in callWithContext
            return func(*args,**kw)
        --- <exception caught here> ---
          File "/home/trac/Projects/Twisted/trunk/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite
            why = getattr(selectable, method)()
          File "/home/trac/Projects/Twisted/trunk/twisted/internet/abstract.py", line 127, in doWrite
            self.producer.resumeProducing()
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/stream.py", line 744, in resumeProducing
            self._doWrite(data)
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/stream.py", line 765, in _doWrite
            self.resumeProducing()
...
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/stream.py", line 744, in resumeProducing
            self._doWrite(data)
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/stream.py", line 762, in _doWrite
            self.consumer.write(data)
          File "/home/trac/Projects/Twisted/trunk/twisted/web2/channel/http.py", line 447, in write
            self.transport.write(data)
          File "/home/trac/Projects/Twisted/trunk/twisted/internet/abstract.py", line 175, in write
            self.startWriting()
        exceptions.RuntimeError: maximum recursion depth exceeded

Change History (5)

comment:1 Changed 16 years ago by Glyph

Milestone: Web2-Gold-Master

comment:2 Changed 14 years ago by camrdale

I just hit this bug as well. The original poster didn't mention it, but the clipped Traceback showed (for me anyway) 484 calls to resumeProducing, and 483 calls to _doWrite.

I think it only occurs in unusual situations where the stream produces small amounts of bytes at a time. In my case, my stream is a modified FileStream that will read a max of 4096 bytes at a time, whereas the default FileStream would read much more using mmap. The number of times the stream.read() gets called increases the chance of hitting this bug.

The obvious and easy fix is to replace one of the calls with a reactor.callLater(0, ...), either at line 744 or 765 of stream.py. Probably the call to resumeProducing in _doWrite at line 765 should be replaced with reactor.callLater(0, self.resumeProducing).

comment:3 in reply to:  2 Changed 14 years ago by hanser

Replying to camrdale:

I just hit this bug as well. The original poster didn't mention it, but the clipped Traceback showed (for me anyway) 484 calls to resumeProducing, and 483 calls to _doWrite.

I think it only occurs in unusual situations where the stream produces small amounts of bytes at a time. In my case, my stream is a modified FileStream that will read a max of 4096 bytes at a time, whereas the default FileStream would read much more using mmap. The number of times the stream.read() gets called increases the chance of hitting this bug.

The obvious and easy fix is to replace one of the calls with a reactor.callLater(0, ...), either at line 744 or 765 of stream.py. Probably the call to resumeProducing in _doWrite at line 765 should be replaced with reactor.callLater(0, self.resumeProducing).

May be the case does not occur so often But for me,

  • it occurred at my first attempt at using web2, when i tried displaying an array

with 300 lines.

  • I discovered it is fixed in the twisted version from cpushare

comment:4 Changed 11 years ago by Jean-Paul Calderone

Description: modified (diff)
Resolution: wontfix
Status: newclosed

Not going to fix this, see #4821. Twisted Web has no similar defect as far as I know, so not retargeting this ticket a the web component.

comment:5 Changed 11 years ago by <automation>

Owner: jknight deleted
Note: See TracTickets for help on using tickets.