[Twisted-Python] memory leaks

[Glyph Lefkowitz]

> On Dec 5, 2016, at 10:18 PM, Amber Hawkie Brown <hawkowl at atleastfornow.net> wrote:
> 
> 
>> On 6 Dec. 2016, at 17:09, Glyph Lefkowitz <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
>> 
>> 
>> twisted.logger._initialBuffer can consume a surprisingly large amount of memory if logging is not initialized
>> 
>> 
>> The way that `twisted.logger` is supposed to work is that at process startup time, the global log observer has a ring buffer for any messages emitted before logging is initialized, and emit those messages to the initial set of log observers passed to `globalLogBeginner.beginLoggingTo`.
>> 
>> The size of this buffer (in `twisted.logger._buffer._DEFAULT_BUFFER_MAXIMUM` is 65535.  This value was selected arbitrarily, probably because somebody (me or wsanchez) thought "huh, yeah, 64k, that's probably a fine number); but of course, that's 64k ''bytes''.
>> 
>> If this were a buffer of actual formatted log messages, of say 200 bytes each, that would be about 13 megabytes, which is maybe an acceptable amount of RAM to spend on a log buffer.
>> 
>> However, it isn't that.  It's a buffer of 64k log ''events'', each of which probably has a `log_logger` and `log_source` set, each of which is an object attached to potentially arbitrary data.  For example, every `Factory` that starts up logs something, which means you're holding on to an instance, and an instance dictionary, and the protocol instance, and the protocol instance dictionary.  Worse yet, any logged ''failures'' might hold on to all the stuff on their stack.
>> 
>> Add it all up and you end up with a log buffer totaling in the hundreds of megabytes, or even gigabytes, once it's full.  In an application that naively uses Twisted without ever initializing logging, this hangs around forever.
>> 
>> This buffer should probably be a ''lot'' smaller, and we might want to emit a warning when it fills up, reminding people that it is ''only polite'' to start up the logging subsystem, even just to explicitly throw logs away.
> 
> 
> I fixed trac: https://twistedmatrix.com/trac/ticket/8936#ticket <https://twistedmatrix.com/trac/ticket/8936#ticket>
Thanks!

By "Fixed" I take it you mean rolled back, via:

trac at dornkirk:~$ ./virtualenv/bin/pip freeze | grep Trac==
Trac==1.0.13

and
https://github.com/twisted-infra/braid/commit/c2d393fd501c6464b1b475eff214cab64f13ee2a

Don't get me wrong, I'm not complaining :). This is certainly the right thing to do for now.  But I'm wondering if you know what's wrong with 1.2 so we can upgrade soonish?

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20161205/9f7bbeb1/attachment-0002.html>