[Twisted-Python] Twisted, twistd and logging

[Tim Allen]

glyph at divmod.com wrote:
> On 02:13 am, tim at commsecure.com.au wrote:
> >Having skim-read ticket 638 that JP pointed to in another reply,
> >now I'm not so sure. It's kind of difficult to for me to understand the
> >current state of play, especially since that ticket predates the merge
> >of seemingly-related ticket 3052, but it *seems* to be a battle between
> >the camps of "twistd's logging is limited, it should be infinitely
> >extensible" and "twistd's logging is limited, applications
> >should be allowed to override it". Personally, I think I lean
> >more toward "it should be infinitely extensible" side - but right now I
> >don't really care about the general case of extensibility, I just want
> >more flexible syslog support.
> 
> Well, let me first say that your understanding of the issues here is 
> remarkably deep and nuanced, despite your modesty :). 

Thank you. :)

> I don't believe there are actually different "camps" here, and there 
> isn't a lot of disagreement.  At least, I believe all of the core 
> developers have essentially the same perspective, with differences in 
> emphasis, not content.
> 
> There are two systems here.  Twisted's logging (i.e. 
> twisted.python.log), and twistd's logging 
> (twisted.internet._twistd_unix, twisted.internet._twistw, 
> twisted.python.syslog, among other things).  Twisted's logging, while it 
> could certainly be improved, is in reasonable shape.  It's extremely 
> simple, and there is a clear and (mostly) well-specified "do anything 
> you want" integration point: addObserver.  While there are some tweaks 
> I'd like to see here, I don't think it's what we're talking about.

True; if I had to make a complaint about t.p.log, it would be that this
system for arbitrary event-sources to communicate with arbitrary
event-observers needs a well-defined interface to facilitate
communication. Currently, the only things an event-observer can rely on
are the 'isError' and 'system' fields being present, and probably
'message' and 'time'. Any observer winds up either depending on extra
fields set by a specific event-source (and hoping that no other
event-source happens to use the same field-names for different
purposes), or having to treat every event as a raw text event -
defeating the point of logging complicated events in the first place.

However! That's probably an entirely separate can of worms from the
actual problem I'm currently supposed to be working on, so I shall
leave it there for now. :)

> The ticket which I believe most clearly explains how to fix this issue 
> is #2751.  The question of "how do we tell where twistd thinks the logs 
> go" is really just a tiny microcosm of "how do we tell what twistd 
> thinks is going on around here".  There is no clear, uniform surface 
> presented to affect twistd's configuration from anywhere, not even 
> internally; let alone in user code.

A kind of document-object-model of application deployment
configuration, then?

> The first application that needs to do this is twistd itself.  The 
> internal APIs of twistd need to be organized such that it is easy to 
> maintain and extend - for example, so that you could do the extension 
> that you are proposing with syslog.  Once these APIs are tested, 
> documented, and clearly factored, adding a plugin system to allow users 
> to replace components of the logging system should be relatively 
> straightforward.
> 
> Work is underway to do all of this; many of the tickets that we're 
> talking about here have already have work underway.

Your most recent comment on #2571 is a lot more optimistic, it sounds
like twistd has most of the stuff it needs already.

If I have some free time, I might poke through the twistd code to see
if I can understand the architectural issues better.

> >At any rate, ticket 638 and other related tickets seem to be high-level
> >conceptual arguments, and I don't feel I have the domain-knowledge
> >required to comment intelligibly on them.
> 
> I hope that this summary has improved your understanding somewhat. 
> Don't let the depth of the issues here scare you; the best way to 
> resolve major architectural headaches like this is one step at a time, 
> responding to one specific use-case at a time.  So, I would recommend 
> that you file a ticket if you haven't already, and simply add syslog 
> facility support to twistd by adding a --syslog-facility command-line 
> option (no, usage.Options won't let you have --foo and --foo=bar options 
> at the same time, sorry...) and a _syslogFacility attribute to the 
> UnixAppLogger object added by the fix to #3052.  You would of course 
> need to add this to twisted.python.syslog as well, but I think the way 
> to do that should be pretty straightforward.

Filed as #3300, with patches and explanations.

> Also, congratulations on volunteering to maintain Twisted's log 
> subsystem!  If you would like to decline, please see David Reid about 
> fighting a bear on your way out.  ;-).

Well, we'll see how I go. ;)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://twistedmatrix.com/pipermail/twisted-python/attachments/20080620/b7bc95e1/attachment.pgp