[Twisted-Python] deferToThread and thrown exceptions break gc - ticket 3853

exarkun at twistedmatrix.com exarkun at twistedmatrix.com
Wed Jun 26 15:07:25 MDT 2013

On 08:15 pm, rutt.4 at osu.edu wrote:
>I am not sure that that trac issue contains a patch that will fix it. 
>most recent patch on that trac issue is for twisted 8.1, and it does 
>apply cleanly to twisted 12.3.0.  When I tried to place the 2 lines 
>form the patch in the most obvious place in the 12.3.0 codebase (before 
>code 'self.waiters.append(ct)') I could not get it to fix the repro I 
>Here's my repro:
>from twisted.internet import reactor
>from twisted.internet.threads import deferToThread
>class SomeClass(object):
>    def __del__(self):
>        print 'SomeClass\'s destructor was called!'

You need to be a little careful here.  Adding an object with a `__del__` 
method is a good way to further confuse garbage collection issues.  For 
this test to be valid, you now also need to prove that this object never 
becomes part of a reference cycle.

It's generally safer to use a weakref with a callback to prove things 
about when objects get collected.

However, I don't think this is a problem in this case.

I did modify your example in another way though:

from twisted.internet import reactor
from twisted.internet.threads import deferToThread

class SomeClass(object):
    def __del__(self):
        print "SomeClass\'s destructor was called!"

def foo():
    sc = SomeClass()
    raise RuntimeError('bah')

def shutmedown(data):
    print 'reactor shutdown happening soon'
    reactor.callLater(1, stop)

def stop():
    print 'reactor shutdown happening now'

def go():
    d = deferToThread(foo)
    d.addCallbacks(shutmedown, shutmedown)
print 'Reactor shutdown'

Notice that now it doesn't shutdown as soon as the thread completes, but 
lets execution continue for a little while longer.

Against trunk at HEAD on Debian Wheezy, I see this behavior:

reactor shutdown happening soon
reactor shutdown happening now
SomeClass's destructor was called!
Reactor shutdown

This still shows there is a problem, since the object is kept alive for 
an arbitrary amount of time after the function in the thread raises the 

Then I inserted just the `sys.exc_clear()` call above the "with 
self._workerState..." statement in threadpool.py in trunk at HEAD.  This 
changes the behavior to:

reactor shutdown happening soon
SomeClass's destructor was called!
reactor shutdown happening now
Reactor shutdown

Notice the destructor is called sooner.  From watching this run, I can 
also say it is called soon after the callback on the deferToThread 
Deferred fires (not, say, after the 1 second delay I inserted).

Whether or not it's possible to have the destructor called *even* sooner 
than this, I don't know.  This behavior does seem like it's good enough 
for most cases though.


More information about the Twisted-Python mailing list