[Twisted-Python] deferToThread and thrown exceptions break gc - ticket 3853

[Benjamin Rutt]

I am not sure that that trac issue contains a patch that will fix it.  The
most recent patch on that trac issue is for twisted 8.1, and it does not
apply cleanly to twisted 12.3.0.  When I tried to place the 2 lines that
form the patch in the most obvious place in the 12.3.0 codebase (before the
code 'self.waiters.append(ct)') I could not get it to fix the repro I have.
 Here's my repro:

from twisted.internet import reactor
from twisted.internet.threads import deferToThread

class SomeClass(object):
    def __del__(self):
        print 'SomeClass\'s destructor was called!'

def foo():
    sc = SomeClass()
    raise RuntimeError('bah')

def shutmedown(data):
    print 'reactor shutdown happening now'
    reactor.stop()

def go():
    d = deferToThread(foo)
    d.addCallbacks(shutmedown, shutmedown)
reactor.callWhenRunning(go)
reactor.run()

and running it:

$ ./twisted-defertothread-bug.py
reactor shutdown happening now
SomeClass's destructor was called!

The bug is present, as the instance of SomeClass only gets gc'd once the
reactor shuts down (taking the thread pool with it), not when the foo()
function terminates.  Obviously I'm using CPython, which uses reference
counting gc, so there is no reason to delay the reclaim of the SomeClass
instance until after the function exit.


On Wed, Jun 26, 2013 at 1:03 PM, Christopher Armstrong <
radix at twistedmatrix.com> wrote:

> On Wed, Jun 26, 2013 at 9:15 AM, Benjamin Rutt <rutt.4 at osu.edu> wrote:
>
>> Hi,
>>
>> I believe I hit http://twistedmatrix.com/trac/ticket/3853 in production
>> this week, in the form of what looked like a resource leak, but turned out
>> to be a case of a resource held for much longer than expected.  (I'm using
>> python 2.7, and twisted 12.3.0).
>>
>> That is, I had a function invoked by deferToThread, that searched a file
>> by memory mapping it using the 'mmap' module.  And if my function could not
>> find what it was looking for in the file, it raised an exception at the
>> end.  I observed that my process was still hanging on to the mmap, due to
>> my local variable representing the mmap (which I presumed would have been
>> gc'd as it went out of scope as part of the throw) not being gc'd when the
>> function ended.  I can work around it in my function by setting that local
>> variable to None right before I throw, but this is python, not C, and in
>> python automatic memory management is assumed.
>>
>> So, +1 for that issue getting fixed.   From the issue it looks like there
>> is a deeper issue in python itself that could fix this, but as the fix
>> discussion is moving at a glacial pace, I say +1 for a twisted workaround,
>> which from the trac discussion, sounds possible.  Thanks,
>>
>>
> Like you said, it looks like there's already a patch that can solve the
> problem. All it needs before it can be reviewed is tests. If you could
> spare some time to write one (or however many are needed) then that would
> increase the ticket's chances for resolution quite a bit :)
>
>
> --
> Christopher Armstrong
> http://radix.twistedmatrix.com/
> http://planet-if.com/
>
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>


-- 
Benjamin Rutt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20130626/b23c0eff/attachment.html>