[Twisted-Python] threadedselectreactor - shutting down the foreign event loop
bob at redivi.com
Mon Jul 18 06:43:25 EDT 2005
On Jul 17, 2005, at 11:54 PM, Toby Dickenson wrote:
> Im looking at moving my PyQt app to threadedselectreactor.
> Previously I was
> keeping twisted and Qt in seperate threads. Avoiding the need to
> cross that
> thread boundary has been a great simplification, but Im having some
> relating to the shutdown of my foreign event loop.
> The examples such as
> handle this by:
> a. Calling reactor.stop() when there is a GUI shutdown request,
> b. Using reactor.addSystemEventTrigger to shut down the foreign
> event loop
> after reactor shutdown.
This is currently the only supported and recommended way.
> That approach seems untenable for me, for several reasons:
> 1. Too invasive. I dont want to pollute Qt code with Twisted details.
> 2. If my app is runing as a COM server then Pythoncom handles the
> main loop,
> not Qt. More pollution.
> 3. The reactor doesnt get shutdown if the foreign event loop shuts
> down for
> some other reason. This leads to deadlocks when the threading
> module tries to
> join the threads managed by the threadpool module, because the
> does know to close those threads.
1. I guess that's too bad, then. It's less invasive than the
alternatives (using an event-loop specific reactor, managing threads
yourself). It's also really the only reliable and correct way to do
it, see below.
2. Each different style of foreign event loop is going to need a
different waker function and shutdown code. That's life. There is
no happy magical cross-platform-toolkit-API to solve this problem.
It's amazing that threadedselectreactor can do what it does so
generically with just two minor integration hooks. In the same way
your application can start up in two different modes, you're going to
need different Twisted integration code.
3. The "pollution" is really quite minimal, you add a couple lines to
defer shutdown of the foreign event loop until the reactor is has
properly shut down. If the foreign event loop shuts down out of your
control, you probably have other issues.
> Im having moderate success with the following alternative structure
> closes the reactor *after* whatever foreign event loop has exited,
> but it
> feels like I am relying on undocumented reactor implementation
You are relying on exactly implementation accidents here. Even with
the implementation accidents that make it appear to work, your
proposed alternative has edge cases that will be rare and impossible
to reproduce (yay threads).
> from twisted.internet.threadedselectreactor import install as
> reactor = twisted_install()
> # Request that the reactor stops itself. Internally the
> # reactor uses self.callLater, so shutdown is not complete
> # when this method returns.
> # One call to reactor.iterate is sufficient to complete
> # All reactor shutdown events.
> # Reactor is now fully shutdown
> Is there a better approach?
The problem is that the waker function (trigger in the above) can be
called at any time -- and it's always called from another thread. In
order to shut down the reactor properly, the waker function needs to
continue to work for the duration of shutdown. reactor.stop() can
take many runloop iterations to complete, as it can involve arbitrary
deferreds and even new network I/O. If the foreign event loop has
shut down already, it probably won't work, and you won't be able to
do a proper shutdown.
It's possible, I guess, use a proxy for the waker function that uses
a thread lock so that you can reliably swap out the real waker at the
right time. However, you must somehow acquire that lock before your
foreign event loop has shut down (which is of the same level of
invasiveness as the recommended solution!) so that it will block the
proxy call until a new waker is installed. In the finally clause,
you'd install a new trivial waker (the apply function would work) and
release that lock so that any pending calls would complete.
With any naive solution, it's possible for a message between the
threadedselectreactor's worker thread and the main thread could get
lost, and then you have a reactor that is in an undefined state. The
waker function must work, or else the reactor won't.
Note that your above code has another problem: it calls
reactor.interleave too early. For the same reason you will have edge
cases during shutdown, you can (in some circumstances) have edge
cases during startup if the waker function does not work properly
before you have entered the foreign event loop! Note that in all of
the examples, reactor.interleave(...) is called while the foreign
event loop has already begun (yay threads, again).
> Should something like my finally block be included as a method of
I doubt it, it's not really possible to make a correct one that's
generic. At least, I can't think of any correct way. Correct
patches accepted :)
> Is it right for reactor.stop to return before the shutdown is
Yes, always, even with the traditional reactors! The reactor can not
nest itself, nor can multiple reactors be used in the same process,
so it wouldn't be possible for reactor.stop() to work correctly if it
More information about the Twisted-Python