[Twisted-Python] Persuading Python's Logging to use twisted.logger

Glyph glyph at twistedmatrix.com
Mon May 9 16:16:17 MDT 2016


> On May 9, 2016, at 1:50 PM, Daniel Sutcliffe <dansut at gmail.com> wrote:
> 
> The project I am working on uses pymodbus which I am sure shares a
> fairly common attribute with many other modules of using Python's
> standard Logging mechanism - a very reasonable choice even for a
> module that supports Twisted, the library can also be used entirely
> synchronously and thus would not want a required dependency of
> Twisted.

Sure, this makes sense.

> It struck me that it would be great to be able to redirect the
> standard logging library to use twisted.logger by some sort of 'Clever
> Monkey Patching' and that this may be a relatively common
> requirement... however after extensive searching, and asking on the
> pymodbus list, I can't find any evidence that such a thing has ever
> been attempted or discussed.

You shouldn't need monkey patching; the standard library logger does have facilities for setting up custom handlers.  We just need to make one.  Honestly it would be great if this were featured within Twisted itself; it would make a great contribution to twisted.logger proper.

> The reverse mechanism of sending twisted.logger's output to the
> standard library is of course handled by the
> twisted.logger.STDLibLogObserver (and similar in twisted legacy
> logging) but the documentation for this even suggests why this is a
> bad idea: 'Warning: specific logging configurations (example: network)
> can lead to this observer blocking.' which seems to me why it would be
> better to attempt this the other way around...

I would very much like to see this :).  The standard library logging mechanism is in common use, and until we can spin it out into a separate project, `twisted.logger´ is unlikely to 

> Am I crazy to even think this? is it just the rambling of
> Python/Twisted newb? Or is there something I'm missing that would make
> this impossible to do generically, and awkward to provide a vague
> recipe of how to do?

No, there's nothing that would make it impossible.  If you want to contribute it.

> I do appreciate that twisted.logger offers a more feature rich
> (structured) API and the Logging API would only be able to provide
> level and text but it would be better than loosing any possibly useful
> log messages from used modules in my mind.

I think you might even be able to claw back some structured information out of the stdlib logging machinery?  LogRecord does have 'args' and 'msg', and we could convert that into a twisted.logger format string.  This would be more expensive than using twisted.logger directly of course, but it would still provide a lot more useful information.

> If anyone can enlighten me I would be most appreciative,

I think just try to write a Handler (<https://docs.python.org/2.7/library/logging.handlers.html#module-logging.handlers>) that converts LogRecord objects to event dictionaries, and then emit them to a <https://twistedmatrix.com/documents/16.1.1/api/twisted.logger.ILogObserver.html> and let us know if you hit any problems.

-glyph





More information about the Twisted-Python mailing list