[Twisted-Python] MemoryError in twisted causes the program to exit, or not

[Antoine Pitrou]

On Fri, 28 Sep 2012 00:09:23 -0700
Glyph <glyph at twistedmatrix.com> wrote:
> 
> Le Sep 27, 2012 à 7:41 PM, Benjamin Rutt <rutt.4 at osu.edu> a écrit :
> 
> > Hi, I am trying to understand what will happen in my long-running twisted server program when the available memory is low. 
> 
> Probably nothing good.
> 
> > If I run the following program with increasing numeric seeds as arguments, 0, 1, 2, 3, 4, … , some of the time the out of memory condition will crash and exit the program right away (when the MemoryError is raised inside twisted core code), and other times it will just carry on with an UnhandledError (if the MemoryError is raised inside my hi() function below).
> 
> Yes.
> 
> Ostensibly, Twisted could deal with this "better" by special-casing MemoryError in top-level error handlers and falling through to exit, rather than attempting to log; but, in cases where only a localized area is experiencing memory pressure and the exception causes cleanup to happen correctly, this would overzealously exit your service even in cases where it could recover.

I agree this would be unwelcome behaviour.

> If you have a better idea for how Twisted might handle this generally though, I'm happy to hear it :).

Calling gc.collect() might improve things, temporarily or not, in some
contexts. Perhaps enough to be able to log the error, so that the
developer isn't left without a clue.

(whether Python itself should try to GC-collect when MemoryError is
raised is an open question)

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net