[Twisted-Python] Persuading Python's Logging to use twisted.logger

[Kevin Conway]

> The reverse mechanism of sending twisted.logger's output to the
> standard library is of course handled by the
> twisted.logger.STDLibLogObserver (and similar in twisted legacy
> logging) but the documentation for this even suggests why this is a
> bad idea: 'Warning: specific logging configurations (example: network)
> can lead to this observer blocking.' which seems to me why it would be
> better to attempt this the other way around...

The warning in the documentation about the potential blocking behaviour of
the StdLibLogObserver is correct, but it does tip a little close to FUD. If
you are using the Syslog or File based log observers then there is no
change in "blocking" behaviour between Twisted and the standard lib. The
Syslog observer and the standard library syslog facility both use exactly
the same standard library syslog module which is known to block under
certain system configurations. The File observer in Twisted and the
standard library both use regular Python file objects that don't leverage
the reactor in any way. Likewise, logging to STDIN/STDERR in both logging
frameworks treats the streams like normal file objects which, when
subjected to back pressure, become a source of blocking time. The only
cases where Twisted and the standard lib differ would be network based
logging.

My recommendation is to use the STdLibLogObserver. This will allow you to
use the twisted.logger in your Twisted code, merge Twisted and non-Twisted
logs into a single stream regardless of their origin, and allow you to tap
into things like the standard lib RotatingFileHandler and file based
configuration system. Should you actually need to pipe logs over the
network it would not be overly difficult to implement a standard lib
logging transport that leverages the reactor when sending data.

On Mon, May 9, 2016 at 5:18 PM Glyph <glyph at twistedmatrix.com> wrote:

>
> > On May 9, 2016, at 1:50 PM, Daniel Sutcliffe <dansut at gmail.com> wrote:
> >
> > The project I am working on uses pymodbus which I am sure shares a
> > fairly common attribute with many other modules of using Python's
> > standard Logging mechanism - a very reasonable choice even for a
> > module that supports Twisted, the library can also be used entirely
> > synchronously and thus would not want a required dependency of
> > Twisted.
>
> Sure, this makes sense.
>
> > It struck me that it would be great to be able to redirect the
> > standard logging library to use twisted.logger by some sort of 'Clever
> > Monkey Patching' and that this may be a relatively common
> > requirement... however after extensive searching, and asking on the
> > pymodbus list, I can't find any evidence that such a thing has ever
> > been attempted or discussed.
>
> You shouldn't need monkey patching; the standard library logger does have
> facilities for setting up custom handlers.  We just need to make one.
> Honestly it would be great if this were featured within Twisted itself; it
> would make a great contribution to twisted.logger proper.
>
> > The reverse mechanism of sending twisted.logger's output to the
> > standard library is of course handled by the
> > twisted.logger.STDLibLogObserver (and similar in twisted legacy
> > logging) but the documentation for this even suggests why this is a
> > bad idea: 'Warning: specific logging configurations (example: network)
> > can lead to this observer blocking.' which seems to me why it would be
> > better to attempt this the other way around...
>
> I would very much like to see this :).  The standard library logging
> mechanism is in common use, and until we can spin it out into a separate
> project, `twisted.logger´ is unlikely to
>
> > Am I crazy to even think this? is it just the rambling of
> > Python/Twisted newb? Or is there something I'm missing that would make
> > this impossible to do generically, and awkward to provide a vague
> > recipe of how to do?
>
> No, there's nothing that would make it impossible.  If you want to
> contribute it.
>
> > I do appreciate that twisted.logger offers a more feature rich
> > (structured) API and the Logging API would only be able to provide
> > level and text but it would be better than loosing any possibly useful
> > log messages from used modules in my mind.
>
> I think you might even be able to claw back some structured information
> out of the stdlib logging machinery?  LogRecord does have 'args' and 'msg',
> and we could convert that into a twisted.logger format string.  This would
> be more expensive than using twisted.logger directly of course, but it
> would still provide a lot more useful information.
>
> > If anyone can enlighten me I would be most appreciative,
>
> I think just try to write a Handler (<
> https://docs.python.org/2.7/library/logging.handlers.html#module-logging.handlers>)
> that converts LogRecord objects to event dictionaries, and then emit them
> to a <
> https://twistedmatrix.com/documents/16.1.1/api/twisted.logger.ILogObserver.html>
> and let us know if you hit any problems.
>
> -glyph
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20160510/41971901/attachment-0002.html>