[Twisted-Python] MemoryError in twisted causes the program to exit, or not

[Glyph]

Le Sep 27, 2012 à 7:41 PM, Benjamin Rutt <rutt.4 at osu.edu> a écrit :

> Hi, I am trying to understand what will happen in my long-running twisted server program when the available memory is low. 

Probably nothing good.

> If I run the following program with increasing numeric seeds as arguments, 0, 1, 2, 3, 4, … , some of the time the out of memory condition will crash and exit the program right away (when the MemoryError is raised inside twisted core code), and other times it will just carry on with an UnhandledError (if the MemoryError is raised inside my hi() function below).

Yes.

Resource exhaustion testing is tough, especially in Python, where literally every function call may (non-deterministically) allocate memory.  MemoryError is an exception that you may see at any time, in any context, without warning.  Most notably you may encounter a MemoryError when attempting to handle a MemoryError and there's not much you can do about it at that point.

Ostensibly, Twisted could deal with this "better" by special-casing MemoryError in top-level error handlers and falling through to exit, rather than attempting to log; but, in cases where only a localized area is experiencing memory pressure and the exception causes cleanup to happen correctly, this would overzealously exit your service even in cases where it could recover.

So: don't exhaust your available memory.  It will cause a lot of unpredictable problems.

If you have a better idea for how Twisted might handle this generally though, I'm happy to hear it :).

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20120928/2155abce/attachment.html>