<font face="Calibri, sans-serif"><div>Hi, I am trying to
understand what will happen in my long-running twisted server program when the available memory is low. <br><br>If I run the
following program with increasing numeric seeds as arguments, 0, 1, 2, 3,
4, … , some of the time
the out of memory condition will crash and exit the program right away (when the MemoryError is raised inside
twisted core code), and other times it will just carry on with an
UnhandledError (if the MemoryError is raised inside my hi() function
below).</div>
<div> </div>
<div>Sample code:</div>
<div> <br>---<br></div>
<div>#!/usr/bin/env python</div>
<div> </div>
<div>import sys, resource, random, traceback</div>
<div> </div>
<div>from twisted.internet import reactor</div>
<div> </div>
<div>resource.setrlimit(resource.RLIMIT_AS, (180 * 1024 * 1024, resource.RLIM_INFINITY))</div>
<div> </div>
<div>data = []</div>
<div>random.seed(int(sys.argv[1]))</div>
<div> </div>
<div>def hi():</div>
<div> data.append('x' * random.randint(1, 10000))</div>
<div> reactor.callLater(0, hi)</div>
<div> </div>
<div>def main():</div>
<div> reactor.callLater(0, hi)</div>
<div> </div>
<div>reactor.callWhenRunning(main)</div>
<div>reactor.run()</div>
<div> ---<br><br></div>
<div>I am using python 2.7.1, twisted 11.0.0.</div>
<div> </div>
<div>Sample runs follow:</div>
<div> </div>
<div>bash$ n=0; while true; do echo "starting rep $n"; ./twisted-oom.py $n; n=$(($n+1)); echo; done</div>
<div>starting rep 0</div>
<div>Traceback (most recent call last):</div>
<div> File “./twisted-oom.py", line 21, in <module></div>
<div> reactor.run()</div>
<div> File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1162, in run</div>
<div> self.mainLoop()</div>
<div> File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1177, in mainLoop</div>
<div> log.err()</div>
<div> File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/python/log.py", line 197, in err</div>
<div> _stuff = failure.Failure()</div>
<div>MemoryError</div>
<div> </div>
<div>starting rep 1</div>
<div>Traceback (most recent call last):</div>
<div> File ". /twisted-oom.py", line 21, in <module></div>
<div> reactor.run()</div>
<div> File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1162, in run</div>
<div> self.mainLoop()</div>
<div> File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1177, in mainLoop</div>
<div> log.err()</div>
<div>MemoryError</div>
<div> </div>
<div>starting rep 2</div>
<div>Unhandled Error</div>
<div>Traceback (most recent call last):</div>
<div> File "./twisted-oom.py", line 21, in <module></div>
<div> reactor.run()</div>
<div> File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1162, in run</div>
<div> self.mainLoop()</div>
<div> File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1171, in mainLoop</div>
<div> self.runUntilCurrent()</div>
<div>--- <exception caught here> ---</div>
<div> File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 793, in runUntilCurrent</div>
<div> call.func(*call.args, **<a href="http://call.kw" target="_blank">call.kw</a>)</div>
<div> File "/home/ruttbe/dev/oneoff/twisted-oom.py", line 14, in hi</div>
<div> data.append('x' * random.randint(1, 10000))</div>
<div>exceptions.MemoryError:</div>
<div> </div>
<div>On the first two runs, the program exited quickly, but on the last rep
("starting rep 2"), it just carried on running with the reactor still
running. It
would be nice if there was a way to make the out of memory behavior
consistent, so I would know that it would either always carry on running
with a
MemoryError stack trace, or always crash out and end the program. Maybe
runUntilCurrent
should not catch MemoryError from user code and should let it propagate
out? Also wondering if there
are other different behaviors in other areas of the code (such as
callbacks into user code such as dataReceived, outReceived,
lineReceived, etc.). i.e. is the observed behavior (that MemoryError in
user code is caught and MemoryError in twisted core is not)
intentional, or an accident?<br></div></font><br clear="all">Thanks,<br>-- <br>Benjamin Rutt<br>
<br>