[Twisted-Python] the good, the log, and the ugly

Glyph glyph at twistedmatrix.com
Fri Mar 30 20:49:04 EDT 2012

On Mar 30, 2012, at 3:23 PM, Brian Warner wrote:

> On 3/25/12 7:11 PM, Glyph wrote:
>> I'm going to try to get started on record my desired behavior for
>> logging as tickets soon though, so Twisted can at least have some really
>> good logging tools /eventually/.  
> While you're thinking about this space, I'll plug the logging tools I
> built into Foolscap (and applications that use it, like Tahoe). I'm sure
> many others have built things like it, but some of the features that
> I've found particularly useful in Tahoe development are:
> * structured log messages, stored as dicts with non-serialized
>   arguments
> * "umids": unique message identifiers, to find the call site
> * size-limited per-(priority*section) circular event buffers
> * parent/child event relationships, for event trees
> * "Incidents": when an UNUSUAL event is logged, a pickle of the event
>   buffers is saved to disk, for later inspection, including events from
>   post-trigger error-recovery code
> * "logport": Foolscap interface for subscribing to event streams
> * "flogtool tail" to connect to remote app and watch the event stream
> * web- or text- based event renderers, with expand/hide-subtree buttons
> * "incident gatherer": subscribes to Incident reports from multiple
>   remote apps, classifies reports them according to cause
> It's inspired by twisted.python.log, of course, and the log.msg()
> interface should look pretty familiar. The Foolscap buy-in may be too
> much for most apps, but it'd be interesting to find a subset that
> doesn't require the networking parts. I tried to figure out how to make
> it fit with Python's stdlib logging module, but eventually gave up (I
> think the biggest issue was needing to return an event number from the
> log.msg() call, for the parent/child stuff).
> http://foolscap.lothar.com/docs/logging.html has details.

Indeed, Foolscap has informed many of my future wishes for Twisted logging, especially its incident reporting and logport features.  (I wasn't aware of UMIDs but I have independently invented the same thing.)

Could we perhaps get some of foolscap logging's core features into Twisted?  We could have an AMP interface for subscribing to log streams that would be slightly more low-fidelity (but, I would guess, higher-performance) than the Foolscap one, but work based on the same core mechanism which foolscap could then subscribe to.

I suspect that this would be useful to foolscap as well, because then the code that foolscap calls into (for example, the web stuff for its UIs) would be using the same logging convention.


More information about the Twisted-Python mailing list