[Twisted-Python] [ANN] structlog – bring context & structure to your logs without boilerplate

Hynek Schlawack hs at ox.cx
Tue Sep 17 10:12:01 MDT 2013


Hi folks,

today, I’ve release the 0.2.0 of my structlog project whose ambition is no less than changing the way we log in Python. :)

And since it sports dedicated support for Twisted <http://www.structlog.org/en/latest/twisted.html> (but is not limited to it, it wraps *any* logger), I’d like to introduce you to it.

Basically, its premise is that events happen in a context and you want to log out both effortlessly.  Nowadays logs are usually consumed by parsers anyway, so regular, easily parsable logs are a good thing™.

This is a nice example of a Twisted application showing off a bit what structlog is capable of:

import sys
import uuid

import structlog
import twisted

from twisted.internet import protocol, reactor

logger = structlog.getLogger()


class Counter(object):
    i = 0

    def inc(self):
        self.i += 1

    def __repr__(self):
        return str(self.i)


class Echo(protocol.Protocol):
    def connectionMade(self):
        self._counter = Counter()
        self._log = logger.new(
            connection_id=str(uuid.uuid4()),
            peer=self.transport.getPeer().host,
            count=self._counter,
        )

    def dataReceived(self, data):
        self._counter.inc()
        log = self._log.bind(data=data)
        self.transport.write(data)
        log.msg('echoed data!')


if __name__ == "__main__":
    structlog.configure(
        processors=[structlog.twisted.EventAdapter()],
        logger_factory=structlog.twisted.LoggerFactory(),
    )
    twisted.python.log.startLogging(sys.stderr)
    reactor.listenTCP(1234, protocol.Factory.forProtocol(Echo))
    reactor.run()


It will give you an output like:

2013-09-17 17:40:23+0200 [-] Log opened.
2013-09-17 17:40:23+0200 [-] Factory starting on 1234
2013-09-17 17:40:23+0200 [-] Starting factory <twisted.internet.protocol.Factory instance at 0x108301488>
2013-09-17 17:40:28+0200 [Echo,0,127.0.0.1] peer='127.0.0.1' count=1 connection_id='4e2ee31c-b3ff-478e-ae06-7b1a492fce45' data='foo\n' event='echoed data!'
2013-09-17 17:40:33+0200 [Echo,0,127.0.0.1] peer='127.0.0.1' count=2 connection_id='4e2ee31c-b3ff-478e-ae06-7b1a492fce45' data='bar\n' event='echoed data!'
2013-09-17 17:40:44+0200 [Echo,1,127.0.0.1] peer='127.0.0.1' count=1 connection_id='3118f673-482b-471f-a206-e8f97f9a9c2c' data='qux\n' event='echoed data!'
2013-09-17 17:40:53+0200 [Echo,0,127.0.0.1] peer='127.0.0.1' count=3 connection_id='4e2ee31c-b3ff-478e-ae06-7b1a492fce45' data='twisted prevails\n' event='echoed data!'

(you can find this and more examples at <http://www.structlog.org/en/latest/examples.html>)

In short:

- It allows you to build context by binding values to loggers. This context is just a dictionary. Once you log an event out (again, arbitrary number of key/value pairs), it gets merged with the context, processed by configurable processor chain and passed to your original logger.
- Your loggers are immutable (by default infidels can use thread local storage) and you’ll get a new one on each binding. Immutable local data is awesome.
- You can define processors that can mangle, filter, and format your log entries.
- Configure once, then just call getLogger (which is a Twisted-friendly alias for get_logger) in regular code.
- Of course there is a JSON renderer built right in – just tell log stash about it and be merry.

*Please* have a look at <http://www.structlog.org/en/0.2.0-0/getting-started.html>, I don’t want to reproduce it here. :)

Credit where credit is due: I picked up the idea of bound loggers from David Reid’s work on otter <https://github.com/rackerlabs/otter/tree/master/otter/log> which in turn was based on ideas by JP (I’ve been told).

You can find everything you need to know at http://www.structlog.org/ – I’d be happy about any constructive feedback and even more so about contributions!

Cheers,
Hynek
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 235 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20130917/6641f382/attachment.pgp>


More information about the Twisted-Python mailing list