I want to install twisted without a c compiler

[zooko]

I was just trying to install twisted on Irby's OLPC. It doesn't have a c compiler. building twisted stops with:

unable to execute gcc: No such file or directory unable to execute gcc: No such file or directory unable to execute gcc: No such file or directory error: Setup script exited with error: command 'gcc' failed with exit status 1

[feisley]

Short of compiling binaries for various platforms, is this possible? There are just some components that need to be compiled to function.

[glyph]

Replying to feisley:

There are just some components that need to be compiled to function.

There are very few extension modules in Twisted:

1. twisted.protocols._c_urlarg, which is a simple optimization for a bottleneck in twisted.web (URL argument decoding) and should be completely optional.
2. twisted.runner.portmap, which allows you to listen on RPC ports with twisted.runner. Nobody uses RPC, and nobody uses twisted.runner. Also, twisted.runner will fail gracefully if it's not installed.
3. twisted.internet.cfsupport, which is a support module for a reactor using the CoreFoundation API on MacOS X. This is nominally for use in MacOS GUI applications, but I have never heard of anyone getting it to work. A while ago I wrote a ticket comment describing that whole mess. At this point, the code should probably just be removed, since nobody who knows how to write mac apps will either maintain or recommend it. At any rate, in the case zooko describes, the module wouldn't even be compiled because it's not a mac.
4. twisted.test.raiser, which is a test support module for a unit test which checks Failure's stack formatting interacting with C extension modules. If you don't compile it you won't be able to run that one test, but that's all; everything still works fine.
5. twisted.python._epoll, which is required for the epoll reactor. The epoll reactor is effectively an optimization for high-volume servers, so you don't need it for Twisted to function.
6. twisted.internet.iocpreactor.iocpsupport, which is required for the I/O Completion Ports reactor on Microsoft Windows servers. Unlike the CF Reactor, this is actually supported, tested, and used by some people. Like the CF Reactor, however, it won't be compiled and can't be used on the platform in question.

So, the only features that you will find missing if you can't compile C extension modules on a UNIX-like platform are the epoll reactor and the ability to listen on RPC ports in twisted.runner. Hardly essential features of Twisted and you should be able to install it without them.

[exarkun]

iocpreactor isn't supported (else it would be here: <http://buildbot.twistedmatrix.com/boxes-supported>)

[exarkun]

I don't think distutils supports this.

exarkun@boson:~$ mkdir distutils-test
exarkun@boson:~$ cd distutils-test/
exarkun@boson:~/distutils-test$ cat > foo.c
int main(int argc, char** argv) {
    return 0;
}
exarkun@boson:~/distutils-test$ cat > setup.py
from distutils.core import setup, Extension
setup(name='foo', version='1.0', ext_modules=[Extension('foo', ['foo.c'])])
exarkun@boson:~/distutils-test$ PATH= /usr/bin/python setup.py install --prefix /tmp/distutils-test
running install
running build
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-i686-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c foo.c -o build/temp.linux-i686-2.5/foo.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
exarkun@boson:~/distutils-test$

Is there some way to tell distutils that it's okay to not build an extension? If not, I think you should file a ticket in the Python tracker asking for this feature and we can use it once it's implemented.

[zooko]

Okay, I opened a ticket for distutils:

http://bugs.python.org/issue4706

That ticket claims that distutils already handles the case that gcc exited with a non-zero return value, but not the case that gcc couldn't be found.

I also added this to the list of issue tickets on my klog.

I can't help thinking that there ought to be a way for Twisted to make this work better before they fix it in Python, they release a new version of Python with the fix, OLPC upgrades to that new version of Python, and Irby upgrades to that new version of OLPC. That chain of events might never actually happen, not least because I'm currently advising Irby to stick with Python 2.5 and not upgrade to newer versions of Python. ;-)

Just to throw out an idea, how about an argument to setup.py that says --please-no-c-modules? It's ugly in various ways, but at least Irby could deploy Twisted onto his OLPC that way.

Another idea would be to provide a subclass of the distutils build command which ignores that exception and converts it into a "normal failure" to build.

[exarkun]

I'd like to see what happens to http://bugs.python.org/issue4706 before trying to come up with a solution of our own.

[exarkun]

Perhaps the resolution to http://bugs.python.org/issue5583 (which issue4706 was closed as a duplicate of) gives us what is necessary here.

http://svn.python.org/view/python/trunk/Doc/distutils/setupscript.rst?r1=70910&r2=70909&pathrev=70910 is mostly the relevant part.

[zooko]

I encountered this issue again recently. I've written some software for internal use at a corporation. They wanted to deploy it to their Windows XP server. I gave them a README.txt saying to run "python ./setup.py install" to install my software along with all of its dependencies, which are:

pyutil 1.4.1, argparse 1.0, lxml 2.2.2, SQLAlchemy 0.5.5, pyodbc 2.1.6, setuptools 0.6c9, txAMQP 0.2, zbase32 1.1.1, Twisted 8.2.0, zope.interface 3.5.2, setuptools 0.6c9

Two of the packages didn't install when they tried to deploy it on the Windows servers. One was txAMQP because it isn't currently reachable from PyPI (see https://bugs.launchpad.net/txamqp/+bug/413691 ). The other was Twisted, because they don't have a compiler on their Windows server. I had to help the Windows server admin download the .exe installer from the twistedmatrix.com web site. When he saw the name of it, he said "Are you sure this is safe?". I assured him that it was a standard Python tool.

So if I understand correctly, once Python 2.7 comes out, then the distutils that comes with that will have this flag with which Twisted can specify that its extensions are optional, and that failure to compile them should not stop the build.

What about for users of older Pythons such as 2.5? (That's the version of Python on this company's Windows server, and it is also the version that I use and support for other cases nowadays.) I saw that the simplejson package has a feature of attempting to compile its extension module and then falling back to pure Python if it fails:

http://simplejson.googlecode.com/svn/trunk/setup.py

I emulated this technique in the setup.py for one of my projects (zbase32) and it worked fine. Perhaps we should do likewise for Twisted?