Opened 11 years ago

Last modified 4 years ago

#3586 enhancement new

— at I want to install twisted without a c compilerVersion 19

Reported by: zooko Owned by:
Priority: normal Milestone:
Component: release management Keywords: build
Cc: zooko@…, soult, Thijs Triemstra, davidsarah, ivank, fijal, anatoly techtonik Branch:
Author:

Description (last modified by Thijs Triemstra)

I was just trying to install twisted on Irby's OLPC. It doesn't have a c compiler. building twisted stops with:

unable to execute gcc: No such file or directory
unable to execute gcc: No such file or directory
unable to execute gcc: No such file or directory
error: Setup script exited with error: command 'gcc' failed with exit status 1

Change History (21)

comment:1 Changed 11 years ago by feisley

Short of compiling binaries for various platforms, is this possible? There are just some components that need to be compiled to function.

comment:2 in reply to:  1 Changed 11 years ago by Glyph

Replying to feisley:

There are just some components that need to be compiled to function.

There are very few extension modules in Twisted:

  1. twisted.protocols._c_urlarg, which is a simple optimization for a bottleneck in twisted.web (URL argument decoding) and should be completely optional.
  2. twisted.runner.portmap, which allows you to listen on RPC ports with twisted.runner. Nobody uses RPC, and nobody uses twisted.runner. Also, twisted.runner will fail gracefully if it's not installed.
  3. twisted.internet.cfsupport, which is a support module for a reactor using the CoreFoundation API on MacOS X. This is nominally for use in MacOS GUI applications, but I have never heard of anyone getting it to work. A while ago I wrote a ticket comment describing that whole mess. At this point, the code should probably just be removed, since nobody who knows how to write mac apps will either maintain or recommend it. At any rate, in the case zooko describes, the module wouldn't even be compiled because it's not a mac.
  4. twisted.test.raiser, which is a test support module for a unit test which checks Failure's stack formatting interacting with C extension modules. If you don't compile it you won't be able to run that one test, but that's all; everything still works fine.
  5. twisted.python._epoll, which is required for the epoll reactor. The epoll reactor is effectively an optimization for high-volume servers, so you don't need it for Twisted to function.
  6. twisted.internet.iocpreactor.iocpsupport, which is required for the I/O Completion Ports reactor on Microsoft Windows servers. Unlike the CF Reactor, this is actually supported, tested, and used by some people. Like the CF Reactor, however, it won't be compiled and can't be used on the platform in question.

So, the only features that you will find missing if you can't compile C extension modules on a UNIX-like platform are the epoll reactor and the ability to listen on RPC ports in twisted.runner. Hardly essential features of Twisted and you should be able to install it without them.

comment:3 Changed 11 years ago by Jean-Paul Calderone

iocpreactor isn't supported (else it would be here: <http://buildbot.twistedmatrix.com/boxes-supported>)

comment:4 Changed 11 years ago by Jean-Paul Calderone

I don't think distutils supports this.

exarkun@boson:~$ mkdir distutils-test
exarkun@boson:~$ cd distutils-test/
exarkun@boson:~/distutils-test$ cat > foo.c
int main(int argc, char** argv) {
    return 0;
}
exarkun@boson:~/distutils-test$ cat > setup.py
from distutils.core import setup, Extension
setup(name='foo', version='1.0', ext_modules=[Extension('foo', ['foo.c'])])
exarkun@boson:~/distutils-test$ PATH= /usr/bin/python setup.py install --prefix /tmp/distutils-test
running install
running build
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-i686-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c foo.c -o build/temp.linux-i686-2.5/foo.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
exarkun@boson:~/distutils-test$ 

Is there some way to tell distutils that it's okay to not build an extension? If not, I think you should file a ticket in the Python tracker asking for this feature and we can use it once it's implemented.

comment:5 Changed 11 years ago by zooko

Okay, I opened a ticket for distutils:

http://bugs.python.org/issue4706

That ticket claims that distutils already handles the case that gcc exited with a non-zero return value, but not the case that gcc couldn't be found.

I also added this to the list of issue tickets on my klog.

I can't help thinking that there ought to be a way for Twisted to make this work better before they fix it in Python, they release a new version of Python with the fix, OLPC upgrades to that new version of Python, and Irby upgrades to that new version of OLPC. That chain of events might never actually happen, not least because I'm currently advising Irby to stick with Python 2.5 and not upgrade to newer versions of Python. ;-)

Just to throw out an idea, how about an argument to setup.py that says --please-no-c-modules? It's ugly in various ways, but at least Irby could deploy Twisted onto his OLPC that way.

Another idea would be to provide a subclass of the distutils build command which ignores that exception and converts it into a "normal failure" to build.

comment:6 Changed 11 years ago by Jean-Paul Calderone

I'd like to see what happens to http://bugs.python.org/issue4706 before trying to come up with a solution of our own.

comment:7 Changed 10 years ago by Jean-Paul Calderone

Perhaps the resolution to http://bugs.python.org/issue5583 (which issue4706 was closed as a duplicate of) gives us what is necessary here.

http://svn.python.org/view/python/trunk/Doc/distutils/setupscript.rst?r1=70910&r2=70909&pathrev=70910 is mostly the relevant part.

comment:8 Changed 10 years ago by zooko

I encountered this issue again recently. I've written some software for internal use at a corporation. They wanted to deploy it to their Windows XP server. I gave them a README.txt saying to run "python ./setup.py install" to install my software along with all of its dependencies, which are:

pyutil 1.4.1, argparse 1.0, lxml 2.2.2, SQLAlchemy 0.5.5, pyodbc 2.1.6, setuptools 0.6c9, txAMQP 0.2, zbase32 1.1.1, Twisted 8.2.0, zope.interface 3.5.2, setuptools 0.6c9

Two of the packages didn't install when they tried to deploy it on the Windows servers. One was txAMQP because it isn't currently reachable from PyPI (see https://bugs.launchpad.net/txamqp/+bug/413691 ). The other was Twisted, because they don't have a compiler on their Windows server. I had to help the Windows server admin download the .exe installer from the twistedmatrix.com web site. When he saw the name of it, he said "Are you sure this is safe?". I assured him that it was a standard Python tool.

So if I understand correctly, once Python 2.7 comes out, then the distutils that comes with that will have this flag with which Twisted can specify that its extensions are optional, and that failure to compile them should not stop the build.

What about for users of older Pythons such as 2.5? (That's the version of Python on this company's Windows server, and it is also the version that I use and support for other cases nowadays.) I saw that the simplejson package has a feature of attempting to compile its extension module and then falling back to pure Python if it fails:

http://simplejson.googlecode.com/svn/trunk/setup.py

I emulated this technique in the setup.py for one of my projects (zbase32) and it worked fine. Perhaps we should do likewise for Twisted?

comment:9 Changed 9 years ago by tarek

Distutils2 has an option to make extensions optional.

Until then you can write a custom build_ext command that handles the compilation errors, and use it in setup.py

Simplest form:

    from distutils.errors import CCompilerError, DistutilsError, CompileError
    from distutils.command.build_ext import build_ext as distutils_build_ext

    class build_ext(distutils_build_ext):

        def build_extension(self, ext):
            try:
                distutils_build_ext.build_extension(self, ext)
            except (CCompilerError, DistutilsError, CompileError), e:
                pass
   

comment:10 Changed 9 years ago by tarek

Oops, just read the simplejson example, it's pretty similar.

comment:11 Changed 9 years ago by zooko

This patch appears to fix it on my manual testing on Mac OS 10.4.

Changed 9 years ago by zooko

Attachment: patch.txt added

comment:12 Changed 9 years ago by zooko

As with many of my intended contributions to Twisted, I have trouble writing a test for this once, since the test probably needs to be run from a shell or a buildbot BuildStep or something rather than from trial.

I *guess* it might be possible to run trial and have a test case which sets up an environment in which it can invoke "python setup.py build" on a Twisted source tree and that no C compiler answers the call, by messing with the PYTHONPATH and the PATH and so on. Shall I try that? Advice would be welcome. I would like this patch to be accepted into Twisted since I'm currently helping yet another person install Twisted on a machine with no compiler...

comment:13 Changed 9 years ago by zooko

Cc: zooko@… added

comment:14 Changed 9 years ago by Jean-Paul Calderone

Would it be possible to have a unit test which fiddled with PATH and invoked the customized build_ext (as opposed to actually running Twisted's setup.py and installing all of actual-Twisted)?

Changed 9 years ago by soult

Attachment: patch2.txt added

comment:15 Changed 9 years ago by soult

Hi there, I tried to install Twisted without a compiler on Windows 7 64bit with Zooko's patch. Using Twisted 10.0.0, I got the following error:

running build_ext error: Unable to find vcvarsall.bat

With Zooko's help I was able to find out that distutils returned DistutilsPlatformError("Unable to find vcvarsall.bat"). After adding an except block for it, it worked like a charm (besides the warning about not using the c extension for speedup).

It would probably be best to make distutils return the correct error instead of a DistutilsPlatformError instead of using this hack.

The attached patch includes the previous patch plus my workaround.

comment:16 Changed 9 years ago by soult

Cc: soult added

comment:17 Changed 9 years ago by zooko

Owner: changed from radix to zooko
Status: newassigned

exarkun: I will try. soult: thanks for the patch.

comment:18 Changed 9 years ago by Thijs Triemstra

Cc: Thijs Triemstra added
Keywords: review added
Owner: zooko deleted
Status: assignednew

Putting the patch up for review.

comment:19 Changed 9 years ago by Thijs Triemstra

Description: modified (diff)
Note: See TracTickets for help on using tickets.