[Twisted-Python] Seeking advice: Coping with heavy log file loads

[Brian Warner]

You might find the logging infrastructure in Foolscap interesting:

 http://foolscap.lothar.com/docs/logging.html

The basic idea is to record log events only in memory, in separate
circular buffers for each priority level. When something "weird"
happens, all the events are bundled into an "Incident File", and then
the incident file is written out (either to disk, or to a gatherer
process over a network). Other tools are then used to reassemble the
pieces of the incident into chronological order and point out where the
gaps are.

I haven't done any specific performance measurements, but I've taken a
busy server and stubbed out the logging calls and didn't see a huge
change in throughput rate. The Foolscap approach gets you useful (and
detailed) information when bad things happen, but throws out most of the
other stuff, so it doesn't touch the disk or make system calls very often.

cheers,
 -Brian (author of Foolscap)