Other wx/twisted methods (was Re: [Twisted-Python] removing unsupported reactors in twisted 2.6: qt, corefoundation, threadedselect, wx)

[glyph at divmod.com]

On Sat, 23 Sep 2006 23:33:11 -0400, Stephen Waterbury <waterbug at pangalactic.us> wrote:

>If I can get that to work (which I haven't had time to play with yet),
>it would seem to offer several advantages:  (1) running wx and twisted
>event loops in completely separate processes, (2) making relatively
>minimal use of threads in the wx process (well-supported by the wx
>libraries) to drive Pyro, and using the usual twisted paradigm in
>the twisted process.  I would appreciate any advice, but I don't see
>any show-stoppers.  I would do the Pyro communications on 127.0.0.1,
>since Pyro has essentially zero security, keeping it completely within
>the local machine.  Twisted would handle all external protocols.

Thanks for the suggestion, Stephen.  I don't think I like this approach though.

The major reason is that it introduces a process boundary between Twisted code and WX code.  In every other toolkit Twisted supports, you can handle GUI events and network events in the same process, in the same thread.   That makes it possible to have individual state machines which can emit output in the form of either GUI updates and writes to streams, and you can similarly receive GUI events or data from the network as input.

Keep in mind that each of the toolkits that WX backends to already *has* either a supported reactor or a proof of concept for Twisted integration: GTK+, the Win32 event loop, and Cococa.  It's not that we can't get network I/O notifications from the platform.  The real problem here is that WX obscures the underlying APIs for integrating sockets.  (I say "obscures" rather than "does not wrap" because the last time I tried to use gtk reactor with a gtk build of wx, the process segfaulted after emitting a pile of dynamic linker errors.  I have no idea what would happen on Windows or the Mac.)  Whatever we're going to do at this point without getting some new code from the WX developers is going to be a workaround.

That said, the mechanism that you propose may be the best way to practically integrate Twisted and WX today.  If it is, I think that it would work better as a separate project, because it is a workaround with a lot of pointy edges and not a real solution.

One major reason I'd like a WX reactor to continue to exist in some form is OSAF's usage of Twisted and WX.  They have a similarly baroque model for handing data back and forth between Twisted code and GUI code, and I'd love to have something to propose to them that was (A) supported by Twisted (B) well tested, and (C) didn't involve lots of potential race conditions, as all shared-state-threading code was likely to do.  I haven't looked at it in a while, but if I remember correctly their separation between Twisted code and GUI code was fairly clean and well-thought-out, but the separation would have been unnecessary if wx had native Twisted integration.

There are other problems with a Pyro-driven Twisted subprocess, which are relevant as the goal here is a well-supported and well-tested path for integration.

Multiprocess communication and process spawning, especially on Windows, is fraught with peril.  We've been working on reliable event-driven process spawning in Twisted for years and years, and only recently have we gotten to the point where the published API works reliably cross-platform.  Given that Twisted would be left out of this loop, I imagine the process control would be rather ad-hoc, so getting the Twisted and WX processes synchronized would be problematic, and extremely hard to unit test.  Although it would be slightly easier if the Twisted proces is 'on top', you still have to inject test code into both the parent and child processes, and have a way to verify the state of both.

The last release of pyro is almost a year old.  That suggests it is not actively maintained.

Pyro doesn't have a buildbot, or (as far as I can tell from the release) any automated tests of its own.  That would make it a wash in terms of tested, robust integration support.

You'd have to re-implement the Pyro protocol on top of Twisted to communicate with it without spawning lots of threads.  That's a bunch of new code which is going to need a home, again, possibly a whole new project.  Even after that's done, Pyro is going to be spawning threads and blocking inside the WX process because the Twisted implementation isn't running there.