[Twisted-Python] Seeking advice: Coping with heavy log file loads
dr.marc.byrd at gmail.com
Fri Apr 9 15:11:04 EDT 2010
Definitely use syslog. To see how much logging is costing you without
changing code, you can simply route to /dev/null via syslog. Btw, w/ local
logging, you may not see it in the cpu usage - more likely in the i/o wait.
In my experience, logging to a remote syslog-ng server over udp has very
little impact on the app, almost like logging to /dev/null (which in turn is
like not logging at all).
Are you by chance on AWS/EC2 ? If so, please contact me to discuss possible
On Fri, Apr 9, 2010 at 11:57 AM, Jason J. W. Williams <
jasonjwwilliams at gmail.com> wrote:
> Hi Paul,
> Is is sporadic? (the CPU burst) If so, you may be seeing the log
> rotation code kicking...and it's probably blocking your app too until
> it completes.
> I've got a couple patches to fix the blocking issue:
> The patch which eliminates old log rotation will probably get rid of
> your CPU problem too.
> On Fri, Apr 9, 2010 at 2:35 AM, Paul Goins <general at vultaire.net> wrote:
> > There may not be a clear answer to this problem, and the answer may or
> > may not be Twisted-specific. I'm just trying to fish for ideas here.
> > The server I'm working on suffers from extremely high CPU time spent in
> > the logging functions. I'm spending between 30 and 40% of all CPU time
> > just in twisted.python.log.msg.
> > Describing the environment:
> > We use DailyLogFiles for file rotation along with the
> > PythonLoggingObserver to allow for logLevel support. (Of course, this
> > gives us significant overhead from Python's complex logging
> > architecture; maybe we should drop this...)
> > On a 1000 request test of this server, we generate 58,274 calls to
> > t.p.log.msg. There's room for cleanup, but there's lots of pressure to
> > keep many of the log calls we have for forensic purposes, so we can only
> > cut this back so far.
> > Anyway, does anyone have any ideas for speedups? Any stories from
> > experience especially? Shots in the dark are welcome. ;-)
> > -
> > Paul Goins
> > _______________________________________________
> > Twisted-Python mailing list
> > Twisted-Python at twistedmatrix.com
> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Twisted-Python