I want to install twisted without a c compiler

[zooko]

I was just trying to install twisted on Irby's OLPC. It doesn't have a c compiler. building twisted stops with:

unable to execute gcc: No such file or directory
unable to execute gcc: No such file or directory
unable to execute gcc: No such file or directory
error: Setup script exited with error: command 'gcc' failed with exit status 1

[feisley]

Short of compiling binaries for various platforms, is this possible? There are just some components that need to be compiled to function.

[glyph]

Replying to feisley:

There are just some components that need to be compiled to function.

There are very few extension modules in Twisted:

1. twisted.protocols._c_urlarg, which is a simple optimization for a bottleneck in twisted.web (URL argument decoding) and should be completely optional.
2. twisted.runner.portmap, which allows you to listen on RPC ports with twisted.runner. Nobody uses RPC, and nobody uses twisted.runner. Also, twisted.runner will fail gracefully if it's not installed.
3. twisted.internet.cfsupport, which is a support module for a reactor using the CoreFoundation API on MacOS X. This is nominally for use in MacOS GUI applications, but I have never heard of anyone getting it to work. A while ago I wrote a ticket comment describing that whole mess. At this point, the code should probably just be removed, since nobody who knows how to write mac apps will either maintain or recommend it. At any rate, in the case zooko describes, the module wouldn't even be compiled because it's not a mac.
4. twisted.test.raiser, which is a test support module for a unit test which checks Failure's stack formatting interacting with C extension modules. If you don't compile it you won't be able to run that one test, but that's all; everything still works fine.
5. twisted.python._epoll, which is required for the epoll reactor. The epoll reactor is effectively an optimization for high-volume servers, so you don't need it for Twisted to function.
6. twisted.internet.iocpreactor.iocpsupport, which is required for the I/O Completion Ports reactor on Microsoft Windows servers. Unlike the CF Reactor, this is actually supported, tested, and used by some people. Like the CF Reactor, however, it won't be compiled and can't be used on the platform in question.

So, the only features that you will find missing if you can't compile C extension modules on a UNIX-like platform are the epoll reactor and the ability to listen on RPC ports in twisted.runner. Hardly essential features of Twisted and you should be able to install it without them.

[exarkun]

iocpreactor isn't supported (else it would be here: < http://buildbot.twistedmatrix.com/boxes-supported>)

[exarkun]

I don't think distutils supports this.

exarkun@boson:~$ mkdir distutils-test
exarkun@boson:~$ cd distutils-test/
exarkun@boson:~/distutils-test$ cat > foo.c
int main(int argc, char** argv) {
    return 0;
}
exarkun@boson:~/distutils-test$ cat > setup.py
from distutils.core import setup, Extension
setup(name='foo', version='1.0', ext_modules=[Extension('foo', ['foo.c'])])
exarkun@boson:~/distutils-test$ PATH= /usr/bin/python setup.py install --prefix /tmp/distutils-test
running install
running build
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-i686-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c foo.c -o build/temp.linux-i686-2.5/foo.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
exarkun@boson:~/distutils-test$ 

Is there some way to tell distutils that it's okay to not build an extension? If not, I think you should file a ticket in the Python tracker asking for this feature and we can use it once it's implemented.

[zooko]

Okay, I opened a ticket for distutils:

 http://bugs.python.org/issue4706

That ticket claims that distutils already handles the case that gcc exited with a non-zero return value, but not the case that gcc couldn't be found.

I also added this to the  list of issue tickets on my klog.

I can't help thinking that there ought to be a way for Twisted to make this work better before they fix it in Python, they release a new version of Python with the fix, OLPC upgrades to that new version of Python, and Irby upgrades to that new version of OLPC. That chain of events might never actually happen, not least because I'm currently advising Irby to stick with Python 2.5 and not upgrade to newer versions of Python. ;-)

Just to throw out an idea, how about an argument to setup.py that says --please-no-c-modules? It's ugly in various ways, but at least Irby could deploy Twisted onto his OLPC that way.

Another idea would be to provide a subclass of the distutils build command which ignores that exception and converts it into a "normal failure" to build.

[exarkun]

I'd like to see what happens to  http://bugs.python.org/issue4706 before trying to come up with a solution of our own.

[exarkun]

Perhaps the resolution to  http://bugs.python.org/issue5583 (which issue4706 was closed as a duplicate of) gives us what is necessary here.

 http://svn.python.org/view/python/trunk/Doc/distutils/setupscript.rst?r1=70910&r2=70909&pathrev=70910 is mostly the relevant part.

[zooko]

I encountered this issue again recently. I've written some software for internal use at a corporation. They wanted to deploy it to their Windows XP server. I gave them a README.txt saying to run "python ./setup.py install" to install my software along with all of its dependencies, which are:

pyutil 1.4.1, argparse 1.0, lxml 2.2.2, SQLAlchemy 0.5.5, pyodbc 2.1.6, setuptools 0.6c9, txAMQP 0.2, zbase32 1.1.1, Twisted 8.2.0, zope.interface 3.5.2, setuptools 0.6c9

Two of the packages didn't install when they tried to deploy it on the Windows servers. One was txAMQP because it isn't currently reachable from PyPI (see  https://bugs.launchpad.net/txamqp/+bug/413691 ). The other was Twisted, because they don't have a compiler on their Windows server. I had to help the Windows server admin download the .exe installer from the twistedmatrix.com web site. When he saw the name of it, he said "Are you sure this is safe?". I assured him that it was a standard Python tool.

So if I understand correctly, once Python 2.7 comes out, then the distutils that comes with that will have this flag with which Twisted can specify that its extensions are optional, and that failure to compile them should not stop the build.

What about for users of older Pythons such as 2.5? (That's the version of Python on this company's Windows server, and it is also the version that I use and support for other cases nowadays.) I saw that the simplejson package has a feature of attempting to compile its extension module and then falling back to pure Python if it fails:

 http://simplejson.googlecode.com/svn/trunk/setup.py

I emulated this technique in the setup.py for one of my projects (zbase32) and it worked fine. Perhaps we should do likewise for Twisted?

[tarek]

Distutils2 has an option to make extensions optional.

Until then you can write a custom build_ext command that handles the compilation errors, and use it in setup.py

Simplest form:

    from distutils.errors import CCompilerError, DistutilsError, CompileError
    from distutils.command.build_ext import build_ext as distutils_build_ext

    class build_ext(distutils_build_ext):

        def build_extension(self, ext):
            try:
                distutils_build_ext.build_extension(self, ext)
            except (CCompilerError, DistutilsError, CompileError), e:
                pass
   

[tarek]

Oops, just read the simplejson example, it's pretty similar.

[zooko]

This patch appears to fix it on my manual testing on Mac OS 10.4.

[zooko]

As with many of my intended contributions to Twisted, I have trouble writing a test for this once, since the test probably needs to be run from a shell or a buildbot BuildStep or something rather than from trial.

I *guess* it might be possible to run trial and have a test case which sets up an environment in which it can invoke "python setup.py build" on a Twisted source tree and that no C compiler answers the call, by messing with the PYTHONPATH and the PATH and so on. Shall I try that? Advice would be welcome. I would like this patch to be accepted into Twisted since I'm currently helping yet another person install Twisted on a machine with no compiler...

[exarkun]

Would it be possible to have a unit test which fiddled with PATH and invoked the customized build_ext (as opposed to actually running Twisted's setup.py and installing all of actual-Twisted)?

[soult]

Hi there, I tried to install Twisted without a compiler on Windows 7 64bit with Zooko's patch. Using Twisted 10.0.0, I got the following error:

running build_ext error: Unable to find vcvarsall.bat

With Zooko's help I was able to find out that distutils returned DistutilsPlatformError("Unable to find vcvarsall.bat"). After adding an except block for it, it worked like a charm (besides the warning about not using the c extension for speedup).

It would probably be best to make distutils return the correct error instead of a DistutilsPlatformError instead of using this hack.

The attached patch includes the previous patch plus my workaround.

[zooko]

exarkun: I will try. soult: thanks for the patch.

[thijs]

Putting the patch up for review.

[exarkun]

This looks like a good start. If we could also add unit tests for the new error handling behavior, then I think we can probably apply the change.

[davidsarah]

Can't this be adequately tested by mocking build_ext.build_ext.build_extensions to raise a CCompilerError and a DistutilsPlatformError?

[davidsarah]

Related to #4612 (about building Twisted on Win7 64-bit with no Microsoft compiler).

[exarkun]

Mocking might be the sanest testing solution. We want to be confident the test actually exercises the code that will be invoked when people run this on the commandline, of course. If there's an API we can invoke (distutils.core.setup perhaps?) which lets us have that confidence, excellent. The other option is to run a child process, which isn't as shiny, but at least obviously exercises the right code.

[glyph]

#4720 looks like a proposal for a solution to this problem. Is it a duplicate?

[davidsarah]

It looks like a duplicate to me, but it also points out that it would be useful to have a way to require particular extensions:

There is a problem with exit codes, though: someone may want the build to stop if a module they need fails to build. I (hastily) propose some kind of --needcexts=x,y,z option, and if any of those modules fail to build, exit with a non-0 code.

Rather than a command-line argument, this could be expressed as an optional setuptools/distutils feature requirement, e.g. "twisted[iocpreactor]" (see #4438 for how to do that).

I wouldn't want this to delay fixing the main issue in this ticket, though.

[ivank]

I applied patch2.txt. setup.py failed to compile portmap, emitted a warning, then didn't try to compile any of the other C extension modules (installation did succeed though). Is this the intended behavior?

[fijal]

The same problem appears when trying to install twisted on top of pypy in virtualenv (multiple people came complaining to #pypy so far). It's not the lack of gcc, but other bits in compilation not working (but the effect is the same and the same patch should fix it)

[techtonik]

How about making an explicit --no-speedups option for the start? Explicit behavior will allow to fail early if compiler for speedups is not detected or installed.

[techtonik]

Added global --no-speedups option. There is a limitation though - this works:

setup.py --no-speedups bdist

This does't:

setup.py bdist --no-speedups

I couldn't find a way to patch distutils.cmd.Command.__getattr__() to pass global option through this check.

[techtonik]

no leftovers

[techtonik]

(removed unused build import)

[exarkun]

I'd rather not add non-standard options to setup.py. This also doesn't help people using easy_install, since they can't pass options to bdist.

[techtonik]

There is a conflicting story for making extension compilation implicitly unnecessary: "I do not want to install Twisted without speedups"

We need to come upon a consensus which scenario is primary.

[glyph]

Replying to exarkun:

I'd rather not add non-standard options to setup.py. This also doesn't help people using easy_install, since they can't pass options to bdist.

We could address the easy_install problem by having someone build binary eggs for each supported platform.

[exarkun]

We could address the easy_install problem by having someone build binary eggs for each supported platform.

But OLPC isn't a supported platform. :)

[zooko]

I see that #5158 has been merged and released in Twisted 11.1.0. That's good! The patch that fixes #5158 does so by checking whether the platform is (probably) CPython in addition to other checks, such as whether the sys/epoll.h header file exists, in order to decide whether to attempt to build certain extensions.

This is what I call the "check if it will work and then do it" approach. An alternative approach that is more general would be the "try it and don't worry if it didn't work" approach.

That approach would hopefully fix both this ticket (build Twisted on a platform with no C compiler) and #4612 (build Twisted on 64-bit Windows with mingw), as well as not regressing #5158 (build Twisted on PyPy). (As fijal mentioned in comment:28.)

My patch2.txt for this ticket implements that approach (copied from  simplejson's setup.py).

I'd like to try to get patch2.txt merged, again. Here's what I think is needed. Please let me know if this is correct:

1. First of all, patch2.txt doesn't have unit tests. At the time, I hadn't yet grokked mockery in its fullness, and without mockery it was complicated to write tests of build logic, where the test code would have to be buildsteps executed by the buildbot instead of unit test methods executes by trial) (see comments 20, 22, 24). #5158 has demonstrated that mockery is indeed a good way to test this sort of thing. I think we can do even better than the unit tests in #5158, by having unit tests which actually invoke build_ext_twisted.

2. As ivank reported in comment:27, the current patch2.txt skips building of all extensions after building of an extension fails. This should probably be changed to attempting to build each extension separately, in case some of them fail but others succeed.

3. Please take a minute to think whether there are problems with the "try it and don't worry if it didn't work" approach. That is: could it be wrong or damaging to attempt to build on a platform where the build will not succeed? Possible problems: 1. it takes up resources such as time or RAM unnecessarily. I don't think this is enough of a problem to worry about. 2. it could have some side-effect such as creating a directory of file during the attempt to build, the presence of which causes some other problem later. I'm not aware of any such problematic side-effects from attempting to build (I'm familiar with Linux, cygwin, Windows-MSVC, Windows-mingw), but if you are please speak up. In any case, remember that there are also potential problems from the "check if it will work and then do it" approach, namely that the "check" step may falsely think that it will work when it won't or think that it won't work when it will.

Okay, if someone will give me a word of encouragement, then I'll proceed to write code for 1 and 2 above (mockery-oriented unit tests and independent attempts per extension).

[exarkun]

At the time, I hadn't yet grokked mockery in its fullness, and without mockery it was complicated to write tests of build logic, where the test code would have to be buildsteps executed by the buildbot instead of unit test methods executes by trial)

Or, the tests could be unit test methods executed by trial (as I mentioned in comment 24). Mocking should be a latter tool to reach for in testing, not an earlier one. What are the obstacles to writing a unit test that actually exercises the code?

Also, another drawback to "try it and don't worry if it didn't work" (aka "easier to ask forgiveness than permission" aka EAFP) is that the build might succeed and produce an extension that is wrong when used (wrong results, crashes, etc).

[zooko]

Replying to exarkun:

What are the obstacles to writing a unit test that actually exercises the code?

The code I'd like to see exercised is build_ext_twisted. The questions that I would like the test to answer about this code are things like "Does it proceed to build extension B even after attempting to build extension A incurred an error?".

But, if when running the test we actually invoke build_ext_twisted and it actually invokes build_ext.build_ext.build_extensions, then a side-effect of running the unit tests will be that the extensions get rebuilt! That sounds potentially troublesome, and we don't need to test that build_ext.build_ext.build_extensions (that's distutils's responsibility, not ours), so I propose to patch out build_ext.build_ext.build_extensions and replace it with a mock object whose job it is to make sure that we pass the correct arguments to build_ext.build_ext.build_extensions (our responsibility, not distutils's).

What do you think?

Also, another drawback to "try it and don't worry if it didn't work" (aka "easier to ask forgiveness than permission" aka EAFP) is that the build might succeed and produce an extension that is wrong when used (wrong results, crashes, etc).

That's a good concern to keep in mind. Do you think that we can rely on the build scripts of our extensions on our platforms to not do that? Or should we abandon this whole approach? Note that the "check if it will work and then do it" approach could also encounter this problem, if the check step decides that it will work when it won't.

[exarkun]

so I propose to patch out build_ext.build_ext.build_extensions and replace it with a mock object whose job it is to make sure that we pass the correct arguments to build_ext.build_ext.build_extensions (our responsibility, not distutils's). What do you think?

I think the difficulty with this is that I don't think any parts of distutils are considered public or stable. A mock runs the risk of becoming desynchronized with the real implementation. (Of course, since Python is effectively unmaintained - or at least in feature freeze - at this point, perhaps this concern should be given less weight.)

However, I do think your concern about actually building Twisted's extension modules is misplaced. The unit test should not try to build Twisted's extension modules. It should try to build some test-provided dummy extensions, at least one of which can be guaranteed to fail to build (perhaps by having a syntax error or including a non-existent header file). Twisted's own extension modules are data for the code in question. Lots of tests exercise code with made up data.

That's a good concern to keep in mind. Do you think that we can rely on the build scripts of our extensions on our platforms to not do that? Or should we abandon this whole approach? Note that the "check if it will work and then do it" approach could also encounter this problem, if the check step decides that it will work when it won't.

A combination of the two makes sense to me. We can skip configurations that we know are invalid so that we don't have to worry about a build inadvertently succeeding. We can also have a safety net, ignoring build failures, so that even things we think should build and work don't prevent build/install from completing if they can't be built.

I think the question of whether it is actually most useful to quietly proceed after build failures hasn't been completely explored, and at some point we may really want build failures to interrupt the normal build/installation process. However I'm happy, for now, to see them not do that, since the current behavior seems to be more troublesome.

[Scramblejams]

Just wanted to say this is still an issue -- I needed to apply Zooko's patch.txt in order to do a --user install of Twisted 12.3.0 on RHEL 6.0 which didn't have the Python dev libraries installed. Thanks Zooko!