[Twisted-Python] Log Rotation Blocks w/ Large Number of Log Files

Maarten ter Huurne maarten at treewalker.org
Tue Mar 16 19:01:00 EDT 2010


On Tuesday 16 March 2010, Jason J. W. Williams wrote:

> Twistd blocks for a long period of time when the number of log files
> built up is large, thereby causing connections to freeze and
> (depending on the length of the rotation) time out.
> 
> Given I've only used Twisted in the past and not contributed to it, I
> was curious if someone could take a look at my changes to
> twisted.python.logfile.LogFile's rotate() method and see if there's
> any reason the changes would be a bad idea.
> 
> What I've attempted to do is defer the rotation of the old log files
> to a thread, and only rotate the current log file in the foreground.

An alternative approach could be to name the log files differently.
For example, use the timestamp of the rotation as a filename suffix:
  twistd.log.2010-03-16.23-56

This is not the traditional way logging is done, but it would get good 
performance with simple code. (It would still need resolution of duplicate 
file names, in case the log rotates very quickly or the clock is moved 
backwards.)

Bye,
		Maarten



More information about the Twisted-Python mailing list