[Twisted-Python] reactor.stop() won't, threads and Queue to blame?

[Andrew Bennetts]

On Sun, Oct 24, 2004 at 10:59:09PM -0400, Clark C. Evans wrote:
[...]
> 
> However, if you want to keep the Queue in the secondary thread,
> you have one problem that is obvious to me:
> 
[...]
> |             print "calling %s(%s,%s)"%(meth.__name__,str(a),str(k))
> |             d.callback(meth(*a,**k))
> |             print "callback done"
[...]
> 
> You seem to be doing d.callback in the secondary thread, rather than
> in the primary thread.  This could be causing some of the problems
> you are experiencing.   It's not customary to use deferreds in any
> other but the main thread.

Yep, that's the problem here.  Change this:
    d.callback(meth(*a,**k))
to this:
    reactor.callFromThread(d.callback, meth(*a, **k))

(Or perhaps less confusingly:
    result = meth(*a, **kw)
    reactor.callFromThread(d.callback, result)
)

> >From a framework perspective, perhaps callback() should raise an 
> error if it is called from anything other than the main thread?
> Or perhaps even I'm not getting it.

There's no reason why Deferreds wouldn't work in another thread, if that's
what you want.  It's just that generally it's not what you want...
Deferreds are used in Twisted to deal with asynchronous operations; but in
non-event loop threads, you'd usually just block.  If for some reason there
were two event-loop threads in the one process, then Deferreds might be
useful in both.

Nothing about Deferreds is at all dependent on the reactor, except for the
ill-conceived setTimeout functionality.  If you want to run a callback chain
in another thread, then Twisted shouldn't stop you (but I would expect you
to very clearly comment your code to explain why, as it would be very
unusual).

This is just a long-winded way of saying that Deferred's implementation
should be completely thread ignorant, even though in practice they're only
used from the main thread.

The real error here wasn't using Deferred.callback in another thread, it was
using reactor.stop in that thread.

-Andrew.