[Twisted-Python] Seeking advice: Coping with heavy log file loads

Kees Bos k.bos at zx.nl
Fri Apr 9 09:53:10 EDT 2010

On Fri, 2010-04-09 at 15:23 +0200, Itamar Turner-Trauring wrote:
> On Fri, 2010-04-09 at 11:49 +0200, Kees Bos wrote:
> > I think you could try to use a custom light weight log function that
> > just queues the log messages (FIFO) and bursts them, say every second,
> > to the log file (e.g. in a thread to use a multicore cpu).
> Or:
> 0. Profile logging system - what exactly is it doing that's using all
> that CPU? If you're unlucky, it's just "lots of Python function calls",
> but maybe it's fixable.
> 1. A log function that sends messages (over e.g. a Unix socket) to
> another process that does the writing. Python GIL means you don't get
> that much benefit from multiple threads.
Duh. Of course.

BTW. syslog from the syslog module on linux just opens a unix socket
to /dev/log and could be used to do just this. (e.g. with log level
LOG_LOCAL0 and configuring your syslog to log facility local0 to a
specific file)

More information about the Twisted-Python mailing list