[Twisted-Python] Seeking advice: Coping with heavy log file loads

[Paul Goins]

There may not be a clear answer to this problem, and the answer may or
may not be Twisted-specific.  I'm just trying to fish for ideas here.

The server I'm working on suffers from extremely high CPU time spent in
the logging functions.  I'm spending between 30 and 40% of all CPU time
just in twisted.python.log.msg.

Describing the environment:

We use DailyLogFiles for file rotation along with the
PythonLoggingObserver to allow for logLevel support.  (Of course, this
gives us significant overhead from Python's complex logging
architecture; maybe we should drop this...)

On a 1000 request test of this server, we generate 58,274 calls to
t.p.log.msg.  There's room for cleanup, but there's lots of pressure to
keep many of the log calls we have for forensic purposes, so we can only
cut this back so far.

Anyway, does anyone have any ideas for speedups?  Any stories from
experience especially?  Shots in the dark are welcome.  ;-)

-
Paul Goins