<div dir="ltr">I am not sure that that trac issue contains a patch that will fix it. The most recent patch on that trac issue is for twisted 8.1, and it does not apply cleanly to twisted 12.3.0. When I tried to place the 2 lines that form the patch in the most obvious place in the 12.3.0 codebase (before the code 'self.waiters.append(ct)') I could not get it to fix the repro I have. Here's my repro:<br>
<div><br></div><div><div style="font-family:Calibri,sans-serif">from twisted.internet import reactor</div><div style="font-family:Calibri,sans-serif">from twisted.internet.threads import deferToThread</div><div style="font-family:Calibri,sans-serif">
</div><div style="font-family:Calibri,sans-serif">class SomeClass(object):</div><div style="font-family:Calibri,sans-serif"> def __del__(self):</div><div style="font-family:Calibri,sans-serif"> print 'SomeClass\'s destructor was called!'</div>
<div style="font-family:Calibri,sans-serif"> </div><div style="font-family:Calibri,sans-serif">def foo():</div><div style="font-family:Calibri,sans-serif"> sc = SomeClass()</div><div style="font-family:Calibri,sans-serif">
raise RuntimeError('bah')</div><div style="font-family:Calibri,sans-serif"> </div><div style="font-family:Calibri,sans-serif">def shutmedown(data):</div><div style="font-family:Calibri,sans-serif"> print 'reactor shutdown happening now'</div>
<div style="font-family:Calibri,sans-serif"> reactor.stop()</div><div style="font-family:Calibri,sans-serif"> </div><div style="font-family:Calibri,sans-serif">def go():</div><div style="font-family:Calibri,sans-serif">
d = deferToThread(foo)</div><div style="font-family:Calibri,sans-serif"> d.addCallbacks(shutmedown, shutmedown)</div><div style="font-family:Calibri,sans-serif">reactor.callWhenRunning(go)</div><div style="font-family:Calibri,sans-serif">
reactor.run()</div><div style="font-family:Calibri,sans-serif"> </div><div style="font-family:Calibri,sans-serif">and running it:</div><div style="font-family:Calibri,sans-serif"><br></div><div style="font-family:Calibri,sans-serif">
$ ./twisted-defertothread-bug.py</div><div style="font-family:Calibri,sans-serif">reactor shutdown happening now</div><div style="font-family:Calibri,sans-serif">SomeClass's destructor was called!</div></div><div style="font-family:Calibri,sans-serif">
<br></div><div style="font-family:Calibri,sans-serif">The bug is present, as the instance of SomeClass only gets gc'd once the reactor shuts down (taking the thread pool with it), not when the foo() function terminates. Obviously I'm using CPython, which uses reference counting gc, so there is no reason to delay the reclaim of the SomeClass instance until after the function exit.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 26, 2013 at 1:03 PM, Christopher Armstrong <span dir="ltr"><<a href="mailto:radix@twistedmatrix.com" target="_blank">radix@twistedmatrix.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="h5">On Wed, Jun 26, 2013 at 9:15 AM, Benjamin Rutt <span dir="ltr"><<a href="mailto:rutt.4@osu.edu" target="_blank">rutt.4@osu.edu</a>></span> wrote:<br>
</div></div><div class="gmail_extra"><div class="gmail_quote"><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hi,</span><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">I believe I hit <a href="http://twistedmatrix.com/trac/ticket/3853" target="_blank">http://twistedmatrix.com/trac/ticket/3853</a> in production this week, in the form of what looked like a resource leak, but turned out to be a case of a resource held for much longer than expected. (I'm using python 2.7, and twisted 12.3.0).</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">That is, I had a function invoked by deferToThread, that searched a file by memory mapping it using the 'mmap' module. And if my function could not find what it was looking for in the file, it raised an exception at the end. I observed that my process was still hanging on to the mmap, due to my local variable representing the mmap (which I presumed would have been gc'd as it went out of scope as part of the throw) not being gc'd when the function ended. I can work around it in my function by setting that local variable to None right before I throw, but this is python, not C, and in python automatic memory management is assumed.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">So, +1 for that issue getting fixed. From the issue it looks like there is a deeper issue in python itself that could fix this, but as the fix discussion is moving at a glacial pace, I say +1 for a twisted workaround, which from the trac discussion, sounds possible. Thanks,</div>
<span><font color="#888888">
<div style="font-family:arial,sans-serif;font-size:13px"><br></div></font></span></div></blockquote><div><br></div></div></div><div>Like you said, it looks like there's already a patch that can solve the problem. All it needs before it can be reviewed is tests. If you could spare some time to write one (or however many are needed) then that would increase the ticket's chances for resolution quite a bit :)</div>
<span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888"><br clear="all"><div><br></div>-- <br>Christopher Armstrong<br><a href="http://radix.twistedmatrix.com/" target="_blank">http://radix.twistedmatrix.com/</a><br>
<a href="http://planet-if.com/" target="_blank">http://planet-if.com/</a><br><br>
</font></span></div></div>
<br>_______________________________________________<br>
Twisted-Python mailing list<br>
<a href="mailto:Twisted-Python@twistedmatrix.com">Twisted-Python@twistedmatrix.com</a><br>
<a href="http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python" target="_blank">http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Benjamin Rutt
</div>