[Twisted-Python] Log Rotation Blocks w/ Large Number of Log Files

Jason J. W. Williams jasonjwwilliams at gmail.com
Tue Mar 16 19:57:08 EDT 2010

True. Technically, you could also just kick it out to syslog and let
logrotate.d do the rotation for you.

The code I put up had a major "concurrency" error in that multiple
rotation threads could be launched and conflict with each other (try
to remove the same file).

Updated the Gist to deal with this: http://gist.github.com/334545


On Tue, Mar 16, 2010 at 5:01 PM, Maarten ter Huurne
<maarten at treewalker.org> wrote:
> On Tuesday 16 March 2010, Jason J. W. Williams wrote:
>> Twistd blocks for a long period of time when the number of log files
>> built up is large, thereby causing connections to freeze and
>> (depending on the length of the rotation) time out.
>> Given I've only used Twisted in the past and not contributed to it, I
>> was curious if someone could take a look at my changes to
>> twisted.python.logfile.LogFile's rotate() method and see if there's
>> any reason the changes would be a bad idea.
>> What I've attempted to do is defer the rotation of the old log files
>> to a thread, and only rotate the current log file in the foreground.
> An alternative approach could be to name the log files differently.
> For example, use the timestamp of the rotation as a filename suffix:
>  twistd.log.2010-03-16.23-56
> This is not the traditional way logging is done, but it would get good
> performance with simple code. (It would still need resolution of duplicate
> file names, in case the log rotates very quickly or the clock is moved
> backwards.)
> Bye,
>                Maarten
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

More information about the Twisted-Python mailing list