[Twisted-Python] Seeking advice: Coping with heavy log file loads

Kees Bos k.bos at zx.nl
Fri Apr 9 05:49:52 EDT 2010

On Fri, 2010-04-09 at 10:35 +0200, Paul Goins wrote:
> There may not be a clear answer to this problem, and the answer may or
> may not be Twisted-specific.  I'm just trying to fish for ideas here.
> The server I'm working on suffers from extremely high CPU time spent in
> the logging functions.  I'm spending between 30 and 40% of all CPU time
> just in twisted.python.log.msg.
> Describing the environment:
> We use DailyLogFiles for file rotation along with the
> PythonLoggingObserver to allow for logLevel support.  (Of course, this
> gives us significant overhead from Python's complex logging
> architecture; maybe we should drop this...)
> On a 1000 request test of this server, we generate 58,274 calls to
> t.p.log.msg.  There's room for cleanup, but there's lots of pressure to
> keep many of the log calls we have for forensic purposes, so we can only
> cut this back so far.
> Anyway, does anyone have any ideas for speedups?  Any stories from
> experience especially?  Shots in the dark are welcome.  ;-)

A request/log ratio of 1:60 is extreme. I would call that a log
generator ;-)

I think you could try to use a custom light weight log function that
just queues the log messages (FIFO) and bursts them, say every second,
to the log file (e.g. in a thread to use a multicore cpu).

More information about the Twisted-Python mailing list