[Twisted-Python] Failure is O(state) vs Exception

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/30/2011 10:39 AM, Andrew Bennetts wrote:
> John Arbash Meinel wrote:
> […]
>>> Just for the record, changing cleanFailure to not do its operation is
>>> likely to lead to big memory leaks. The second thing you did, setting
>>> self.tb to None, makes more sense to speed things up.
>>
>> Right. I agree that it is a better solution. I was wondering about
>> having a whitelist of Exception classes that aren't considered
>> interesting enough to keep a traceback.
> 
> I've filed <http://twistedmatrix.com/trac/ticket/5011> about this issue,
> and I have a patch that takes an alternative approach: don't capture
> tracebacks when Deferred callbacks raise exceptions unless
> defer.setDebugging(True) has been called.
> 
> Another possibility might be to stop Failure from capturing all the
> frame details (i.e. the locals and globals) by default (or at least when
> invoked by Deferred._runCallbacks), as that's probably the bulk of the
> expense and would be very rarely used.
> 
> -Andrew.

I think walking the frames and copying the dicts is also expensive. That
is what the bug you linked to me was about. (First, walking everything
and using __dict__.copy() was a bit expensive, and second that the
safe_repr() calls were turning 1GB strings into a new 1GB+ string.)

The one other step that I think we need, is that 'maybeDeferred' also
always traps into a Failure object, and we'd want that to check
Deferred.debug first.

I do wonder if Failure should just be checking Deferred.debug before
automatically including a traceback. I'm not really sure about logical
layering of twisted modules, though. Certainly 'twisted.python.failure'
seems a lower layer than 'twisted.internet.defer'.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2S7uAACgkQJdeBCYSNAAPS9ACfdGmt9td2jqOFDZYY6mJy/6DN
UiMAnRF0w9pID0WMP0MMGwh9LSQN9RqM
=g3j7
-----END PGP SIGNATURE-----