[Twisted-Python] MemoryError in twisted causes the program to exit, or not
Benjamin Rutt
rutt.4 at osu.edu
Thu Sep 27 22:41:36 EDT 2012
Hi, I am trying to understand what will happen in my long-running twisted
server program when the available memory is low.
If I run the following program with increasing numeric seeds as arguments,
0, 1, 2, 3, 4, … , some of the time the out of memory condition will crash
and exit the program right away (when the MemoryError is raised inside
twisted core code), and other times it will just carry on with an
UnhandledError (if the MemoryError is raised inside my hi() function below).
Sample code:
---
#!/usr/bin/env python
import sys, resource, random, traceback
from twisted.internet import reactor
resource.setrlimit(resource.RLIMIT_AS, (180 * 1024 * 1024,
resource.RLIM_INFINITY))
data = []
random.seed(int(sys.argv[1]))
def hi():
data.append('x' * random.randint(1, 10000))
reactor.callLater(0, hi)
def main():
reactor.callLater(0, hi)
reactor.callWhenRunning(main)
reactor.run()
---
I am using python 2.7.1, twisted 11.0.0.
Sample runs follow:
bash$ n=0; while true; do echo "starting rep $n"; ./twisted-oom.py $n;
n=$(($n+1)); echo; done
starting rep 0
Traceback (most recent call last):
File “./twisted-oom.py", line 21, in <module>
reactor.run()
File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
self.mainLoop()
File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1177, in mainLoop
log.err()
File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/python/log.py",
line 197, in err
_stuff = failure.Failure()
MemoryError
starting rep 1
Traceback (most recent call last):
File ". /twisted-oom.py", line 21, in <module>
reactor.run()
File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
self.mainLoop()
File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1177, in mainLoop
log.err()
MemoryError
starting rep 2
Unhandled Error
Traceback (most recent call last):
File "./twisted-oom.py", line 21, in <module>
reactor.run()
File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1162, in run
self.mainLoop()
File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 1171, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File
"/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py",
line 793, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/ruttbe/dev/oneoff/twisted-oom.py", line 14, in hi
data.append('x' * random.randint(1, 10000))
exceptions.MemoryError:
On the first two runs, the program exited quickly, but on the last rep
("starting rep 2"), it just carried on running with the reactor still
running. It would be nice if there was a way to make the out of memory
behavior consistent, so I would know that it would either always carry on
running with a MemoryError stack trace, or always crash out and end the
program. Maybe runUntilCurrent should not catch MemoryError from user code
and should let it propagate out? Also wondering if there are other
different behaviors in other areas of the code (such as callbacks into user
code such as dataReceived, outReceived, lineReceived, etc.). i.e. is the
observed behavior (that MemoryError in user code is caught and MemoryError
in twisted core is not) intentional, or an accident?
Thanks,
--
Benjamin Rutt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20120927/d6b78e3d/attachment.htm
More information about the Twisted-Python
mailing list