[Twisted-Python] Failure is O(state) vs Exception
John Arbash Meinel
john at arbash-meinel.com
Wed Mar 30 04:50:40 EDT 2011
-----BEGIN PGP SIGNED MESSAGE-----
On 03/30/2011 10:39 AM, Andrew Bennetts wrote:
> John Arbash Meinel wrote:
>>> Just for the record, changing cleanFailure to not do its operation is
>>> likely to lead to big memory leaks. The second thing you did, setting
>>> self.tb to None, makes more sense to speed things up.
>> Right. I agree that it is a better solution. I was wondering about
>> having a whitelist of Exception classes that aren't considered
>> interesting enough to keep a traceback.
> I've filed <http://twistedmatrix.com/trac/ticket/5011> about this issue,
> and I have a patch that takes an alternative approach: don't capture
> tracebacks when Deferred callbacks raise exceptions unless
> defer.setDebugging(True) has been called.
> Another possibility might be to stop Failure from capturing all the
> frame details (i.e. the locals and globals) by default (or at least when
> invoked by Deferred._runCallbacks), as that's probably the bulk of the
> expense and would be very rarely used.
I think walking the frames and copying the dicts is also expensive. That
is what the bug you linked to me was about. (First, walking everything
and using __dict__.copy() was a bit expensive, and second that the
safe_repr() calls were turning 1GB strings into a new 1GB+ string.)
The one other step that I think we need, is that 'maybeDeferred' also
always traps into a Failure object, and we'd want that to check
I do wonder if Failure should just be checking Deferred.debug before
automatically including a traceback. I'm not really sure about logical
layering of twisted modules, though. Certainly 'twisted.python.failure'
seems a lower layer than 'twisted.internet.defer'.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
-----END PGP SIGNATURE-----
More information about the Twisted-Python