CFReactor Doesn't Work (like, at all)

[indigo]

At  http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor.html#auto11 it says, "use cfreactor". Then, it says something about a Cocoa directory of examples. Lacking an obvious link or other location, one can find Twisted-2.4.0/TwistedCore-2.4.0/doc/examples/threadedselect/Cocoa/SimpleWebClient in the source tarball. But, SimpleWebClient doesn't use cfreactor. It uses threadedselectreactor, which is undocumented. Some on IRC say threadedselectreactor "kinda replaces cfreactor", but I am confused.

[glyph]

This is indeed a problem.

The story goes something like this. It's from memory, so apologies if I get anything wrong.

Bob (etrepum) half-finished cfReactor and then left it alone for a while. He discovered several limitations in it, stopped using it, then wrote threadedselectreactor (or "TSR") to replace it because integrating with corefoundation is apparently really hard. Considering that he is the only person who has done any significant work on cfReactor, that makes it effectively unmaintained.

Bob went on to declare that TSR should replace all the other GUI event loop reactors (particularly wxreactor). Itamar implemented a wxreactor based on TSR, but in the process discovered some very nasty limitations of using threads to control the main loop. The usual fun stuff about signal interaction and what-not, if I recall correctly. There was some mailing list noise about this, but I can't find a link right now.

No buildbots are currently covering the TS or CF reactors, or the WX reactor based on TSR. That makes them effectively unmaintained. Apple's recent contribution of a buildbot would help if we had anyone currently on the team (Bob hasn't contributed in a while) who had a mac and the know-how to make it work properly.

Personally, I haven't figured out how to run the cfreactor tests yet, and threadedselectreactor fails hundreds of tests when I run it on a mac. I haven't tried other platforms recently, but I assume they're the same.

If you've got some corefoundation experience and have some suggestions as to what we should be doing with CFReactor, I'd be glad to hear about it. However, I have no idea what you should do if you're developing a Cocoa+Twisted application. I don't like its design, but Bob knows more about the mac than I do, and he says TSR is a good choice. So maybe use that, but be aware of the general situation w.r.t. failing tests, etc.

I really would like Twisted to be an excellent platform for Mac-native network applications, so I'd be very glad if someone stepped up to resolve this situation, but I don't even use a mac on a regular basis any more. My development these days is almost entirely server-based, which makes the mac an irrelevant platform. Sorry I couldn't have a more useful answer here.

Although I can tell this story, I don't even know how to update the docs, because I only have a series of vague impressions and remembered mailing list discussions to go on, nothing really technical to explain.

[rikyu]

CFReactor implementation using TSR

[rikyu]

I'm adding this here because I don't know what else to do with it. This is a fully working CFReactor using ThreadedSelectReactor. It's really trivial, since TSR was all but designed for CF.

To elaborate on the issues mentioned by glyph, there are architectural choices in TSR that prevent most of its use-cases from being fully tested. The biggest problem is that TSR allows the native toolkit's event loop to dictate when the reactor will stop.

This has the most obvious implications when dealing with Trial. Trial assumes that reactor.run() will return after another thread calls reactor.crash(). This can't be done in Cocoa (so far as I can tell), because anything I've found that will make PyObjCTools.AppHelper.runEventLoop() stop will terminate the entire process.

This seems to also be the issue for t.i.wxreactor.

[exarkun]

(In [26670]) Branching to 'cfreactor-1833'

[exarkun]

(In [26671]) Add Phil Christensen's new cfreactor with a few trivial tweaks

refs #1833

[exarkun]

I've replaced the old cfreactor with the new one from rikyu's patch. I also changed the web client demo to use cfreactor again, so I think the docs may be right now. I haven't tested the new reactor, as I have no OS X machine handy at the moment.

[glyph]

You should remove the extension definition for cfsupport in twisted/topfiles/setup.py.

I don't think this works in trunk or in the branch. All things being equal I'm inclined to say let's just merge the damn thing and be done with it, since there's less code to maintain and the brokenness is similar, but... well. Here are the inconsistencies in behavior I discovered upon cursory inspection on an actual mac.

In trunk, the "Quit" menu item in "Twistzilla" works. In the branch it doesn't.

In trunk, 'trial -r cf twisted' crashes in twisted.test.test_dirdbm.DirDbmTestCase.testRebuildInteraction after a crash dialog pops up; the error message on the console is "Trace/BPT trap".

In the branch, 'trial -r cf twisted' crashes in twisted.conch.test.test_manhole.ManholeLoopbackStdio.testClassDefinition. The error message on the console is "Python[2153:727] No Info.plist file in application bundle or no NSPrincipalClass in the Info.plist file, exiting".

In trunk, 'twistd -r cf -n web --path .' works basically as expected.

In the branch, 'twistd -r cf -n web --path .' exits instantly. This is the only thing that makes me think that maybe leaving the existing cfreactor around for a while would be a good idea. Here's the log:

$ twistd -r cf -n web --path .
2009-04-13 16:06:50-0400 [-] Log opened.
2009-04-13 16:06:50-0400 [-] twistd 8.2.0+r26691 (/System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python 2.5.1) starting up.
2009-04-13 16:06:50-0400 [-] reactor class: twisted.internet.cfreactor.CFReactor.
2009-04-13 16:06:50-0400 [-] twisted.web.server.Site starting on 8080
2009-04-13 16:06:50-0400 [-] Starting factory <twisted.web.server.Site instance at 0x130bda0>
2009-04-13 16:06:50.240 Python[2188:727] No Info.plist file in application bundle or no NSPrincipalClass in the Info.plist file, exiting

It also occurs to me that ThreadedSelectReactor is badly factored, and it should be something more like ThreadedMultiplexingReactor; you should be able to wrap it around different multiplexing mechanisms, not just select. That's for another ticket, of course.

[exarkun]

rikyu, do you have any thoughts on the Info.plist issues?

[rikyu]

Unfortunately, I do not. As it was, I was never able to get trial to run at all; I wish I had kept better notes on the specific symptoms, but as I recall it couldn't even perform a single test because of the reactor.stop()/.crash()/.run() interactions with AppKit (mentioned in my comment from 2007-10-23).

I did some digging, and this seems to be another issue that is based around the architectural assumptions made by the OS X AppKit framework. The most common reason for this error in other scenarios are attempts to run OS X application binaries directly, rather than through the bundle interface. That would make sense, considering we're effectively trying to do the same thing here; create and run a Cocoa application without a bundle.

In a limited inspection of the test cases mentioned by glyph, it seems that the error doesn't occur until the reactor is started. However, the one test case that did pass successfully was twisted.conch.test.test_agent, but I can't ascertain why, since it seems to have to use the reactor as well.

The best solution I can think of to this issue would be to build a very small PyObjC app just for running trial. I don't like that idea much, but if that gets cfreactor to pass the tests, then I wouldn't complain. I will try to make an attempt at this soon, but (of course) the usual disclaimers of time and workload apply ;-)

[thijs]

Renaming the ticket because it doesn't (only) seem to be documentation related.

[glyph]

#2812 was a duplicate of this.

[glyph]

(In [30231]) Fix cfreactor. Remove cfsupport.

Author: glyph

Reviewer: exarkun

Fixes: #1833

Fixes: #2556

Fixes: #4652

This change significantly improves cfreactor. It is a rewrite from our broken, unmaintained 'cfsupport' pyrex/C module (which has been removed) to use PyObjC's CoreFoundation and CFNetwork support. This fixes several very serious bugs, including many segfaults.

This also changes the testing status of cfreactor. Previously it received no coverage at all from the buildbots. While this change does not add any unique test code, it adds the CFReactor to all tests which use 'reactorbuilder'. These tests would previously fail (several with segfaults) and they now all pass. The result of this is that most of the direct reactor tests are invoked on our existing OS X buildbot. It's still not 100% supported, since there is still no buildbot running under the cfreactor via 'trial -r cf'. A few tests still fail intermittently in that configuration, but I believe they are all bugs in the tests, rather than in the reactor.

This change also allows Twisted to be built with the 'clang' compiler from the LLVM project. Since 'cfsupport' was the only code that had problems with that compiler, its removal fixes the issue.

(This change also includes several unrelated flymake issues.)

[glyph]

Fix some bookkeeping.