[Twisted-Python] Failure is O(state) vs Exception

Christopher Armstrong radix at twistedmatrix.com
Mon Mar 28 11:02:15 EDT 2011

On Mon, Mar 28, 2011 at 9:44 AM, John Arbash Meinel
<john at arbash-meinel.com>wrote:

> Hash: SHA1
> I'm doing some performance testing of one of our Twisted applications.
> And what I came across was a surprising amount of time being spent in
> twisted.python.failure.Failure.__getstate__
> So under profiling, we spent 408ms in __getstate__. I then changed
> Failure.cleanFailure to just 'return', and I saw a real-world
> improvement of ~480ms down to about 240ms. I then restored cleanFailure,
> but changed Failure.__init__ to always set 'tb=None' before it does
> anything. And in that case the time went down to 180ms. (And when I dug
> into it, 150ms of that is time spent waiting for an XMLRPC response.)
Just for the record, changing cleanFailure to not do its operation is likely
to lead to big memory leaks. The second thing you did, setting self.tb to
None, makes more sense to speed things up.

> I'm wondering if there is a tasteful way to disable Traceback processing
> in a production machine. I realize you never know when you are going to
> need the state in order to figure out what went wrong. But it is costing
> 2-5x runtime speed. (The other answer is to write all of our code to
> avoid Deferred.addErrback...)

I hate to mention it (since it's unlikely I'd be able to work on it), but I
wonder how much a C extension would improve this without going the full step
of throwing out the debug data.

Christopher Armstrong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://twistedmatrix.com/pipermail/twisted-python/attachments/20110328/73e41df0/attachment.htm 

More information about the Twisted-Python mailing list