I want to install twisted without a c compiler

[zooko]

I was just trying to install twisted on Irby's OLPC. It doesn't have a c compiler. building twisted stops with:

unable to execute gcc: No such file or directory
unable to execute gcc: No such file or directory
unable to execute gcc: No such file or directory
error: Setup script exited with error: command 'gcc' failed with exit status 1

[feisley]

Short of compiling binaries for various platforms, is this possible? There are just some components that need to be compiled to function.

[glyph]

Replying to feisley:

There are just some components that need to be compiled to function.

There are very few extension modules in Twisted:

1. twisted.protocols._c_urlarg, which is a simple optimization for a bottleneck in twisted.web (URL argument decoding) and should be completely optional.
2. twisted.runner.portmap, which allows you to listen on RPC ports with twisted.runner. Nobody uses RPC, and nobody uses twisted.runner. Also, twisted.runner will fail gracefully if it's not installed.
3. twisted.internet.cfsupport, which is a support module for a reactor using the CoreFoundation API on MacOS X. This is nominally for use in MacOS GUI applications, but I have never heard of anyone getting it to work. A while ago I wrote a ticket comment describing that whole mess. At this point, the code should probably just be removed, since nobody who knows how to write mac apps will either maintain or recommend it. At any rate, in the case zooko describes, the module wouldn't even be compiled because it's not a mac.
4. twisted.test.raiser, which is a test support module for a unit test which checks Failure's stack formatting interacting with C extension modules. If you don't compile it you won't be able to run that one test, but that's all; everything still works fine.
5. twisted.python._epoll, which is required for the epoll reactor. The epoll reactor is effectively an optimization for high-volume servers, so you don't need it for Twisted to function.
6. twisted.internet.iocpreactor.iocpsupport, which is required for the I/O Completion Ports reactor on Microsoft Windows servers. Unlike the CF Reactor, this is actually supported, tested, and used by some people. Like the CF Reactor, however, it won't be compiled and can't be used on the platform in question.

So, the only features that you will find missing if you can't compile C extension modules on a UNIX-like platform are the epoll reactor and the ability to listen on RPC ports in twisted.runner. Hardly essential features of Twisted and you should be able to install it without them.

[exarkun]

iocpreactor isn't supported (else it would be here: < http://buildbot.twistedmatrix.com/boxes-supported>)

[exarkun]

I don't think distutils supports this.

exarkun@boson:~$ mkdir distutils-test
exarkun@boson:~$ cd distutils-test/
exarkun@boson:~/distutils-test$ cat > foo.c
int main(int argc, char** argv) {
    return 0;
}
exarkun@boson:~/distutils-test$ cat > setup.py
from distutils.core import setup, Extension
setup(name='foo', version='1.0', ext_modules=[Extension('foo', ['foo.c'])])
exarkun@boson:~/distutils-test$ PATH= /usr/bin/python setup.py install --prefix /tmp/distutils-test
running install
running build
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-i686-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c foo.c -o build/temp.linux-i686-2.5/foo.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
exarkun@boson:~/distutils-test$ 

Is there some way to tell distutils that it's okay to not build an extension? If not, I think you should file a ticket in the Python tracker asking for this feature and we can use it once it's implemented.

[zooko]

Okay, I opened a ticket for distutils:

 http://bugs.python.org/issue4706

That ticket claims that distutils already handles the case that gcc exited with a non-zero return value, but not the case that gcc couldn't be found.

I also added this to the  list of issue tickets on my klog.

I can't help thinking that there ought to be a way for Twisted to make this work better before they fix it in Python, they release a new version of Python with the fix, OLPC upgrades to that new version of Python, and Irby upgrades to that new version of OLPC. That chain of events might never actually happen, not least because I'm currently advising Irby to stick with Python 2.5 and not upgrade to newer versions of Python. ;-)

Just to throw out an idea, how about an argument to setup.py that says --please-no-c-modules? It's ugly in various ways, but at least Irby could deploy Twisted onto his OLPC that way.

Another idea would be to provide a subclass of the distutils build command which ignores that exception and converts it into a "normal failure" to build.

[exarkun]

I'd like to see what happens to  http://bugs.python.org/issue4706 before trying to come up with a solution of our own.

[exarkun]

Perhaps the resolution to  http://bugs.python.org/issue5583 (which issue4706 was closed as a duplicate of) gives us what is necessary here.

 http://svn.python.org/view/python/trunk/Doc/distutils/setupscript.rst?r1=70910&r2=70909&pathrev=70910 is mostly the relevant part.

[zooko]

I encountered this issue again recently. I've written some software for internal use at a corporation. They wanted to deploy it to their Windows XP server. I gave them a README.txt saying to run "python ./setup.py install" to install my software along with all of its dependencies, which are:

pyutil 1.4.1, argparse 1.0, lxml 2.2.2, SQLAlchemy 0.5.5, pyodbc 2.1.6, setuptools 0.6c9, txAMQP 0.2, zbase32 1.1.1, Twisted 8.2.0, zope.interface 3.5.2, setuptools 0.6c9

Two of the packages didn't install when they tried to deploy it on the Windows servers. One was txAMQP because it isn't currently reachable from PyPI (see  https://bugs.launchpad.net/txamqp/+bug/413691 ). The other was Twisted, because they don't have a compiler on their Windows server. I had to help the Windows server admin download the .exe installer from the twistedmatrix.com web site. When he saw the name of it, he said "Are you sure this is safe?". I assured him that it was a standard Python tool.

So if I understand correctly, once Python 2.7 comes out, then the distutils that comes with that will have this flag with which Twisted can specify that its extensions are optional, and that failure to compile them should not stop the build.

What about for users of older Pythons such as 2.5? (That's the version of Python on this company's Windows server, and it is also the version that I use and support for other cases nowadays.) I saw that the simplejson package has a feature of attempting to compile its extension module and then falling back to pure Python if it fails:

 http://simplejson.googlecode.com/svn/trunk/setup.py

I emulated this technique in the setup.py for one of my projects (zbase32) and it worked fine. Perhaps we should do likewise for Twisted?

[tarek]

Distutils2 has an option to make extensions optional.

Until then you can write a custom build_ext command that handles the compilation errors, and use it in setup.py

Simplest form:

    from distutils.errors import CCompilerError, DistutilsError, CompileError
    from distutils.command.build_ext import build_ext as distutils_build_ext

    class build_ext(distutils_build_ext):

        def build_extension(self, ext):
            try:
                distutils_build_ext.build_extension(self, ext)
            except (CCompilerError, DistutilsError, CompileError), e:
                pass
   

[tarek]

Oops, just read the simplejson example, it's pretty similar.

[zooko]

This patch appears to fix it on my manual testing on Mac OS 10.4.

[zooko]

As with many of my intended contributions to Twisted, I have trouble writing a test for this once, since the test probably needs to be run from a shell or a buildbot BuildStep or something rather than from trial.

I *guess* it might be possible to run trial and have a test case which sets up an environment in which it can invoke "python setup.py build" on a Twisted source tree and that no C compiler answers the call, by messing with the PYTHONPATH and the PATH and so on. Shall I try that? Advice would be welcome. I would like this patch to be accepted into Twisted since I'm currently helping yet another person install Twisted on a machine with no compiler...

[exarkun]

Would it be possible to have a unit test which fiddled with PATH and invoked the customized build_ext (as opposed to actually running Twisted's setup.py and installing all of actual-Twisted)?

[soult]

Hi there, I tried to install Twisted without a compiler on Windows 7 64bit with Zooko's patch. Using Twisted 10.0.0, I got the following error:

running build_ext error: Unable to find vcvarsall.bat

With Zooko's help I was able to find out that distutils returned DistutilsPlatformError("Unable to find vcvarsall.bat"). After adding an except block for it, it worked like a charm (besides the warning about not using the c extension for speedup).

It would probably be best to make distutils return the correct error instead of a DistutilsPlatformError instead of using this hack.

The attached patch includes the previous patch plus my workaround.

[zooko]

exarkun: I will try. soult: thanks for the patch.

[thijs]

Putting the patch up for review.

[exarkun]

This looks like a good start. If we could also add unit tests for the new error handling behavior, then I think we can probably apply the change.

[davidsarah]

Can't this be adequately tested by mocking build_ext.build_ext.build_extensions to raise a CCompilerError and a DistutilsPlatformError?

[davidsarah]

Related to #4612 (about building Twisted on Win7 64-bit with no Microsoft compiler).

[exarkun]

Mocking might be the sanest testing solution. We want to be confident the test actually exercises the code that will be invoked when people run this on the commandline, of course. If there's an API we can invoke (distutils.core.setup perhaps?) which lets us have that confidence, excellent. The other option is to run a child process, which isn't as shiny, but at least obviously exercises the right code.