[Twisted-Python] Seeking advice: Coping with heavy log file loads
Jason J. W. Williams
jasonjwwilliams at gmail.com
Fri Apr 9 14:57:31 EDT 2010
Is is sporadic? (the CPU burst) If so, you may be seeing the log
rotation code kicking...and it's probably blocking your app too until
I've got a couple patches to fix the blocking issue:
The patch which eliminates old log rotation will probably get rid of
your CPU problem too.
On Fri, Apr 9, 2010 at 2:35 AM, Paul Goins <general at vultaire.net> wrote:
> There may not be a clear answer to this problem, and the answer may or
> may not be Twisted-specific. I'm just trying to fish for ideas here.
> The server I'm working on suffers from extremely high CPU time spent in
> the logging functions. I'm spending between 30 and 40% of all CPU time
> just in twisted.python.log.msg.
> Describing the environment:
> We use DailyLogFiles for file rotation along with the
> PythonLoggingObserver to allow for logLevel support. (Of course, this
> gives us significant overhead from Python's complex logging
> architecture; maybe we should drop this...)
> On a 1000 request test of this server, we generate 58,274 calls to
> t.p.log.msg. There's room for cleanup, but there's lots of pressure to
> keep many of the log calls we have for forensic purposes, so we can only
> cut this back so far.
> Anyway, does anyone have any ideas for speedups? Any stories from
> experience especially? Shots in the dark are welcome. ;-)
> Paul Goins
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
More information about the Twisted-Python