[Twisted-Python] MemoryError in twisted causes the program to exit, or not

Benjamin Rutt rutt.4 at osu.edu
Thu Sep 27 22:41:36 EDT 2012


Hi, I am trying to understand what will happen in my long-running twisted
server program when the available memory is low.

If I run the following program with increasing numeric seeds as arguments,
0, 1, 2, 3, 4, … , some of the time the out of memory condition will crash
and exit the program right away (when the MemoryError is raised inside
twisted core code), and other times it will just carry on with an
UnhandledError (if the MemoryError is raised inside my hi() function below).

Sample code:

---
#!/usr/bin/env python

import sys, resource, random, traceback

from twisted.internet         import reactor

resource.setrlimit(resource.RLIMIT_AS, (180 * 1024 * 1024,
resource.RLIM_INFINITY))

data = []
random.seed(int(sys.argv[1]))

def hi():
    data.append('x' * random.randint(1, 10000))
    reactor.callLater(0, hi)

def main():
    reactor.callLater(0, hi)

reactor.callWhenRunning(main)
reactor.run()
 ---

I am using python 2.7.1, twisted 11.0.0.

Sample runs follow:

bash$ n=0; while true; do echo "starting rep $n"; ./twisted-oom.py $n;
n=$(($n+1)); echo; done
starting rep 0
Traceback (most recent call last):
  File “./twisted-oom.py", line 21, in <module>
    reactor.run()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
    self.mainLoop()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1177, in mainLoop
    log.err()
  File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/python/log.py",
line 197, in err
    _stuff = failure.Failure()
MemoryError

starting rep 1
Traceback (most recent call last):
  File ". /twisted-oom.py", line 21, in <module>
    reactor.run()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
    self.mainLoop()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1177, in mainLoop
    log.err()
MemoryError

starting rep 2
Unhandled Error
Traceback (most recent call last):
  File "./twisted-oom.py", line 21, in <module>
    reactor.run()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
    self.mainLoop()
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1171, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 793, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/home/ruttbe/dev/oneoff/twisted-oom.py", line 14, in hi
    data.append('x' * random.randint(1, 10000))
exceptions.MemoryError:

On the first two runs, the program exited quickly, but on the last rep
("starting rep 2"), it just carried on running with the reactor still
running.  It would be nice if there was a way to make the out of memory
behavior consistent, so I would know that it would either always carry on
running with a MemoryError stack trace, or always crash out and end the
program.  Maybe runUntilCurrent should not catch MemoryError from user code
and should let it propagate out?  Also wondering if there are other
different behaviors in other areas of the code (such as callbacks into user
code such as dataReceived, outReceived, lineReceived, etc.).  i.e. is the
observed behavior (that MemoryError in user code is caught and MemoryError
in twisted core is not) intentional, or an accident?

Thanks,
-- 
Benjamin Rutt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20120927/d6b78e3d/attachment.htm 


More information about the Twisted-Python mailing list