Opened 5 years ago

Last modified 5 years ago

#7841 enhancement new

LimitedHistoryObserver uses too much memory

Reported by: burak arslan Owned by:
Priority: normal Milestone:
Component: core Keywords: logging
Cc: Branch: log-booyah-6750-5
branch-diff, diff-cov, branch-cov, buildbot
Author:

Description

While looking at the memory usage of one of the long-running processes I'm responsible for, I noticed that LimitedHistoryObserver was responsible for holding on to ~30k dict instances. While not strictly a bug (it's under the hard-coded limit of 65536 entries) this should be re-thought as having all this data in memory is not necessary at all.

I'd solve this by having the LimitedHistoryObserver flush all its data to a user-supplied sink destroying itself once one becomes available.

Change History (1)

comment:1 Changed 5 years ago by burak arslan

For completeness, here's a workaround that implements my suggested solution: https://github.com/plq/neurons/blob/417836a729706f2d1449109ccbb7d4a510227050/neurons/daemon/config.py#L623

Note: See TracTickets for help on using tickets.