[Twisted-Python] Twisted, twistd and logging

[glyph at divmod.com]

On 02:13 am, tim at commsecure.com.au wrote:
>Having skim-read ticket 638 that JP pointed to in another reply,
>now I'm not so sure. It's kind of difficult to for me to understand the
>current state of play, especially since that ticket predates the merge
>of seemingly-related ticket 3052, but it *seems* to be a battle between
>the camps of "twistd's logging is limited, it should be infinitely
>extensible" and "twistd's logging is limited, applications
>should be allowed to override it". Personally, I think I lean
>more toward "it should be infinitely extensible" side - but right now I
>don't really care about the general case of extensibility, I just want
>more flexible syslog support.

Well, let me first say that your understanding of the issues here is 
remarkably deep and nuanced, despite your modesty :).  I hope that 
you'll stick around to help us take the whole logging system in Twisted 
to the next level...

I don't believe there are actually different "camps" here, and there 
isn't a lot of disagreement.  At least, I believe all of the core 
developers have essentially the same perspective, with differences in 
emphasis, not content.

There are two systems here.  Twisted's logging (i.e. 
twisted.python.log), and twistd's logging 
(twisted.internet._twistd_unix, twisted.internet._twistw, 
twisted.python.syslog, among other things).  Twisted's logging, while it 
could certainly be improved, is in reasonable shape.  It's extremely 
simple, and there is a clear and (mostly) well-specified "do anything 
you want" integration point: addObserver.  While there are some tweaks 
I'd like to see here, I don't think it's what we're talking about.

By contrast, twistd's logging, as the module names I mentioned above may 
indicate, is a squirrely mess of private and undocumented code, ad-hoc 
relationships, and so on.

The ticket which I believe most clearly explains how to fix this issue 
is #2751.  The question of "how do we tell where twistd thinks the logs 
go" is really just a tiny microcosm of "how do we tell what twistd 
thinks is going on around here".  There is no clear, uniform surface 
presented to affect twistd's configuration from anywhere, not even 
internally; let alone in user code.

The problem with the way you phrased the disagreement is the extreme 
vagueness of the word "application".  It really just means "some code". 
The reason you have picked up on opposition to the "applications should 
be allowed to override it" idea is that there is a tendency, when 
creating applications, to customize every aspect of the framework just 
because you can; to create a monolithic, non-cooperative chunk of code 
that just has a mainpoint and sets everything up for you.  An 
application, as in some code that uses Twisted to provide a network 
service, should be written in such a way that it is relatively agnostic 
about logging.  Some users might want to set it up to use syslog, some 
might want to get it in the logging module; some might want something 
really weird like Growl notifications or dbus messages.  Ergo, 
applications should not generally touch logging, no matter what our 
logging APIs look like.  This is what I'm talking about with the "it 
invites abuse" comment near the bottom of #638.

However, some "applications" *are* logging.  For example, if you wanted 
to do write some crazy log monitor that integrates with twistd process, 
you need two things: the "application" needs to allow the framework to 
handle logging, not muck with it itself (i.e. the thing I just said) and 
the framework needs to provide an interface for *your* application (i.e. 
the log monitor) to get in there and call some nice, well-documented 
APIs to reconfigure the way twistd has set up its logging.

The first application that needs to do this is twistd itself.  The 
internal APIs of twistd need to be organized such that it is easy to 
maintain and extend - for example, so that you could do the extension 
that you are proposing with syslog.  Once these APIs are tested, 
documented, and clearly factored, adding a plugin system to allow users 
to replace components of the logging system should be relatively 
straightforward.

Work is underway to do all of this; many of the tickets that we're 
talking about here have already have work underway.
>At any rate, ticket 638 and other related tickets seem to be high-level
>conceptual arguments, and I don't feel I have the domain-knowledge
>required to comment intelligibly on them.

I hope that this summary has improved your understanding somewhat. 
Don't let the depth of the issues here scare you; the best way to 
resolve major architectural headaches like this is one step at a time, 
responding to one specific use-case at a time.  So, I would recommend 
that you file a ticket if you haven't already, and simply add syslog 
facility support to twistd by adding a --syslog-facility command-line 
option (no, usage.Options won't let you have --foo and --foo=bar options 
at the same time, sorry...) and a _syslogFacility attribute to the 
UnixAppLogger object added by the fix to #3052.  You would of course 
need to add this to twisted.python.syslog as well, but I think the way 
to do that should be pretty straightforward.

If you feel the need to get fancy and add more features to t.p.syslog, 
make the factoring of the log configuration object finer-grained, or add 
a plugin system for twistd for log sinks so you can write your own 
syslog support, feel free!  However, ambitious and possibly 
controversial refactorings like that are a lot easier to talk about (and 
get done) when we can look at how they will improve the implementation 
of some existing feature, not for a speculative benefit.  So in any 
event you probably want to do the simple syslog facility addition first.

Also, congratulations on volunteering to maintain Twisted's log 
subsystem!  If you would like to decline, please see David Reid about 
fighting a bear on your way out.  ;-).