[Twisted-Python] Garbage in logfiles?

[Conor Sayres]

Hi All,

I'm working for an astro observatory and we are upgrading our telescope control software.  We are swapping to twisted under the hood, and I have run into an issue regarding logging.

I have set up twisted logging to record all communication between our devices.  When we simulate our system with a high frequency log rollover rate (~5 seconds), many of the logs contain only garbage.  Here is an example of a few lines in a given bad log file:

3230 3133 2d31 302d 3136 2031 303a 3131
3a34 352d 3037 3030 205b 2d5d 2044 6576
436d 6428 5354 4154 5553 290a 3230 3133
2d31 302d 3136 2031 303a 3131 3a34 352d
3037 3030 205b 2d5d 2044 6576 436d 6428
5354 4154 5553 290a 3230 3133 2d31 302d

Here is an example of a few lines from a good log file:

2013-10-16 10:11:39-0700 [_SocketProtocol,client] Galil Reply(XQ#STATUS)
2013-10-16 10:11:39-0700 [-] Galil Reply(: 1,  1,  1,  1,  1 axis homed)
2013-10-16 10:11:39-0700 [-] To All Users(1 1 i axisHomed=1, 1, 1, 1, 1)
2013-10-16 10:11:39-0700 [-] Galil Reply(000000000, 000000000, 000000000, 000000000, 000000000 commanded position)
2013-10-16 10:11:39-0700 [-] To All Users(1 1 i cmdMount=0, 0, 0, 0, 0)
2013-10-16 10:11:39-0700 [_SocketProtocol,client] MirrorCtrl Reply(1 1 i axisHomed=1, 1, 1, 1, 1)
2013-10-16 10:11:39-0700 [_SocketProtocol,client] Galil Reply(XQ#STATUS)

I haven't been able to reproduce the problem in a simpler framework than our simulation.  I am triggering the rollover by directly calling LogFile.rollover on a timer.  The problem also exists if I define a small file size limit for automatic rollover and let twisted do it automatically.

Sound familiar to anyone out there?

Cheers,
Conor