[Twisted-Python] LimitedHistoryLogObserver is considered harmful ?

[Glyph]

> On Apr 2, 2015, at 4:03 PM, Burak Arslan <burak.arslan at arskom.com.tr> wrote:
> 
> Hi,
> 
> I happen to have the new logging hotness(tm) running in production as
> the backend to python's own logging module in one of the daemons that
> I'm responsible for.

Thanks for trying out this code and helping us validate it!

> I also have dowser[1] integrated with this one so tryng to optimize
> memory usage is almost fun :) So I noticed one day
> LimitedHistoryObserver was responsible for holding on to ~30k dict
> instances. I fixed this with [2]. Resident set size with 48hrs uptime is
> now down ~300M   thanks to that function (and is flat).

That's... a lot of dict instances :).  I guess not a very "limited" history observer ;-).

> That's a bit too much for my tastes so I thought maybe you can
> reconsider that 64 * 1024 entry limit or choose to get rid of the
> default observer automatically once the global observer has a proper
> user-supplied sink.

Can you file this bug on the Twisted tracker so we don't lose it?  I am pretty sure that 300M of memory for buffered logs is not something anybody really wants.

> Best regards,
> Burak
> 
> PS: Neurons will be a web framework that ties a lot of stuff like
> sqlalchemy, spyne and twisted together. Currently it does only two
> things quite well: Forking and rendering html forms :)
> 
> [1] https://github.com/plq/neurons/tree/master/neurons/daemon/dowser
> [2]
> https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050/neurons/daemon/config.py#L623 <https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050/neurons/daemon/config.py#L623>

Awesome, I can't wait to see the results.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20150402/70274616/attachment-0002.html>