[Twisted-Python] LimitedHistoryLogObserver is considered harmful ?

[Burak Arslan]

Hi,

I happen to have the new logging hotness(tm) running in production as
the backend to python's own logging module in one of the daemons that
I'm responsible for.

I also have dowser[1] integrated with this one so tryng to optimize
memory usage is almost fun :) So I noticed one day
LimitedHistoryObserver was responsible for holding on to ~30k dict
instances. I fixed this with [2]. Resident set size with 48hrs uptime is
now down ~300M   thanks to that function (and is flat).

That's a bit too much for my tastes so I thought maybe you can
reconsider that 64 * 1024 entry limit or choose to get rid of the
default observer automatically once the global observer has a proper
user-supplied sink.

Best regards,
Burak

PS: Neurons will be a web framework that ties a lot of stuff like
sqlalchemy, spyne and twisted together. Currently it does only two
things quite well: Forking and rendering html forms :)

[1] https://github.com/plq/neurons/tree/master/neurons/daemon/dowser
[2]
https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050/neurons/daemon/config.py#L623