Opened 7 years ago

Last modified 7 years ago

#7873 enhancement new

Clean `Failure` objects in way that preserves traceback

Reported by: Itamar Turner-Trauring Owned by:
Priority: normal Milestone:
Component: core Keywords:
Cc: Branch:
Author:

Description

Currently Failure.cleanFailure, which gets called by Deferred, will store a string version of the traceback. The reason we do this is to prevent massive memory leaks caused by circular references: the frame objects in the tracebacks can refer to objects that in turn refer to the Failure. Unfortunately this means re-raising the exception is missing the original traceback, which is unfortunate when debugging in some cases (see e.g. https://github.com/itamarst/crochet/issues/77).

We can do better, I think.

  • On Python 3.4 there is a traceback.clean_frames function that cleans the frames out safely. We should definitely use this.
  • On older versions of Python one can clear the locals on a frame... but this can break running code that is still using the frame. This does suggest a potential solution, though: for each frame do a sys.getrefcount. If the refcount is 2 (the attribute lookup's copy plus the reference in the traceback object) then the locals can be cleared since no other code is using that frame. A frame whose locals are in use would would eventually require calling Failure.cleanFailure multiple times to do further breaking of circular references, so perhaps this is not workable.
  • Another potential workaround is removing the reliance on __del__ from Failure. There would still be potential for circular references causing memory leaks, though.

Change History (2)

comment:1 Changed 7 years ago by Itamar Turner-Trauring

One could also go with the refcount option and fallback to existing solution if any of the frames have refcount higher than 2. So at least one would preserve tracebacks some of the time.

comment:2 Changed 7 years ago by Itamar Turner-Trauring

Jean-Paul points out that refcounts won't work on PyPy.

Note: See TracTickets for help on using tickets.