[Twisted-Python] Seeking advice: Coping with heavy log file loads

Itamar Turner-Trauring itamar at itamarst.org
Fri Apr 9 09:23:01 EDT 2010

On Fri, 2010-04-09 at 11:49 +0200, Kees Bos wrote:

> I think you could try to use a custom light weight log function that
> just queues the log messages (FIFO) and bursts them, say every second,
> to the log file (e.g. in a thread to use a multicore cpu).


0. Profile logging system - what exactly is it doing that's using all
that CPU? If you're unlucky, it's just "lots of Python function calls",
but maybe it's fixable.

1. A log function that sends messages (over e.g. a Unix socket) to
another process that does the writing. Python GIL means you don't get
that much benefit from multiple threads.

2. Rewrite log function in C... and then maybe run it in another thread.

More information about the Twisted-Python mailing list