[Twisted-Python] Seeking advice: Coping with heavy log file loads

Jason J. W. Williams jasonjwwilliams at gmail.com
Fri Apr 9 14:57:31 EDT 2010

Hi Paul,

Is is sporadic? (the CPU burst) If so, you may be seeing the log
rotation code kicking...and it's probably blocking your app too until
it completes.

I've got a couple patches to fix the blocking issue:


The patch which eliminates old log rotation will probably get rid of
your CPU problem too.


On Fri, Apr 9, 2010 at 2:35 AM, Paul Goins <general at vultaire.net> wrote:
> There may not be a clear answer to this problem, and the answer may or
> may not be Twisted-specific.  I'm just trying to fish for ideas here.
> The server I'm working on suffers from extremely high CPU time spent in
> the logging functions.  I'm spending between 30 and 40% of all CPU time
> just in twisted.python.log.msg.
> Describing the environment:
> We use DailyLogFiles for file rotation along with the
> PythonLoggingObserver to allow for logLevel support.  (Of course, this
> gives us significant overhead from Python's complex logging
> architecture; maybe we should drop this...)
> On a 1000 request test of this server, we generate 58,274 calls to
> t.p.log.msg.  There's room for cleanup, but there's lots of pressure to
> keep many of the log calls we have for forensic purposes, so we can only
> cut this back so far.
> Anyway, does anyone have any ideas for speedups?  Any stories from
> experience especially?  Shots in the dark are welcome.  ;-)
> -
> Paul Goins
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

More information about the Twisted-Python mailing list