[Twisted-Python] Failure is O(state) vs Exception

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/28/2011 05:02 PM, Christopher Armstrong wrote:
> On Mon, Mar 28, 2011 at 9:44 AM, John Arbash Meinel
> <john at arbash-meinel.com <mailto:john at arbash-meinel.com>> wrote:
> 
>     -----BEGIN PGP SIGNED MESSAGE-----
>     Hash: SHA1
> 
>     I'm doing some performance testing of one of our Twisted applications.
>     And what I came across was a surprising amount of time being spent in
>     twisted.python.failure.Failure.__getstate__
> 
>     So under profiling, we spent 408ms in __getstate__. I then changed
>     Failure.cleanFailure to just 'return', and I saw a real-world
>     improvement of ~480ms down to about 240ms. I then restored cleanFailure,
>     but changed Failure.__init__ to always set 'tb=None' before it does
>     anything. And in that case the time went down to 180ms. (And when I dug
>     into it, 150ms of that is time spent waiting for an XMLRPC response.)
> 
> 
> Just for the record, changing cleanFailure to not do its operation is
> likely to lead to big memory leaks. The second thing you did, setting
> self.tb to None, makes more sense to speed things up.
> 

Right. I agree that it is a better solution. I was wondering about
having a whitelist of Exception classes that aren't considered
interesting enough to keep a traceback.

>  
> 
>     I'm wondering if there is a tasteful way to disable Traceback processing
>     in a production machine. I realize you never know when you are going to
>     need the state in order to figure out what went wrong. But it is costing
>     2-5x runtime speed. (The other answer is to write all of our code to
>     avoid Deferred.addErrback...)
> 
> 
> 
> I hate to mention it (since it's unlikely I'd be able to work on it),
> but I wonder how much a C extension would improve this without going the
> full step of throwing out the debug data.

A large portion of the time spent is in "safe_repr", which presumably is
turning all 30k objects into 'str' representations. I would imagine that
a lot of this is redundant across state. Meaning that the objects up at
the top frame probably don't change very often.


I think part of the issue is that when you have an unhandled exception,
that's when you really want the traceback, but you only know that long
after the traceback is gone.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2QpSwACgkQJdeBCYSNAANHhwCgj3J5a6/FgV1n/O2Zwm8U9pm7
eTgAoKWemYqgHnvMwnCrQA+uFZK/+Zeo
=x5Xm
-----END PGP SIGNATURE-----