Opened 6 years ago

Last modified 11 months ago

#3586 enhancement new

I want to install twisted without a c compiler

Reported by: zooko Owned by: zooko
Priority: normal Milestone:
Component: release management Keywords: build
Cc: zooko@…, david+twisted@…, thijs, davidsarah, ivank-twisted-bugs@…, fijal, techtonik@… Branch:
Author: Launchpad Bug:

Description (last modified by thijs)

I was just trying to install twisted on Irby's OLPC. It doesn't have a c compiler. building twisted stops with:

unable to execute gcc: No such file or directory
unable to execute gcc: No such file or directory
unable to execute gcc: No such file or directory
error: Setup script exited with error: command 'gcc' failed with exit status 1

Attachments (5)

patch.txt (1.3 KB) - added by zooko 4 years ago.
patch2.txt (1.9 KB) - added by soult 4 years ago.
3586.no-speedups.patch (1.7 KB) - added by techtonik 4 years ago.
3586.no-speedups.2.patch (1.4 KB) - added by techtonik 4 years ago.
no leftovers
3586.no-speedups.3.patch (1.3 KB) - added by techtonik 4 years ago.
(removed unused build import)

Download all attachments as: .zip

Change History (46)

comment:1 follow-up: Changed 6 years ago by feisley

Short of compiling binaries for various platforms, is this possible? There are just some components that need to be compiled to function.

comment:2 in reply to: ↑ 1 Changed 6 years ago by glyph

Replying to feisley:

There are just some components that need to be compiled to function.

There are very few extension modules in Twisted:

  1. twisted.protocols._c_urlarg, which is a simple optimization for a bottleneck in twisted.web (URL argument decoding) and should be completely optional.
  2. twisted.runner.portmap, which allows you to listen on RPC ports with twisted.runner. Nobody uses RPC, and nobody uses twisted.runner. Also, twisted.runner will fail gracefully if it's not installed.
  3. twisted.internet.cfsupport, which is a support module for a reactor using the CoreFoundation API on MacOS X. This is nominally for use in MacOS GUI applications, but I have never heard of anyone getting it to work. A while ago I wrote a ticket comment describing that whole mess. At this point, the code should probably just be removed, since nobody who knows how to write mac apps will either maintain or recommend it. At any rate, in the case zooko describes, the module wouldn't even be compiled because it's not a mac.
  4. twisted.test.raiser, which is a test support module for a unit test which checks Failure's stack formatting interacting with C extension modules. If you don't compile it you won't be able to run that one test, but that's all; everything still works fine.
  5. twisted.python._epoll, which is required for the epoll reactor. The epoll reactor is effectively an optimization for high-volume servers, so you don't need it for Twisted to function.
  6. twisted.internet.iocpreactor.iocpsupport, which is required for the I/O Completion Ports reactor on Microsoft Windows servers. Unlike the CF Reactor, this is actually supported, tested, and used by some people. Like the CF Reactor, however, it won't be compiled and can't be used on the platform in question.

So, the only features that you will find missing if you can't compile C extension modules on a UNIX-like platform are the epoll reactor and the ability to listen on RPC ports in twisted.runner. Hardly essential features of Twisted and you should be able to install it without them.

comment:3 Changed 6 years ago by exarkun

iocpreactor isn't supported (else it would be here: <http://buildbot.twistedmatrix.com/boxes-supported>)

comment:4 Changed 6 years ago by exarkun

I don't think distutils supports this.

exarkun@boson:~$ mkdir distutils-test
exarkun@boson:~$ cd distutils-test/
exarkun@boson:~/distutils-test$ cat > foo.c
int main(int argc, char** argv) {
    return 0;
}
exarkun@boson:~/distutils-test$ cat > setup.py
from distutils.core import setup, Extension
setup(name='foo', version='1.0', ext_modules=[Extension('foo', ['foo.c'])])
exarkun@boson:~/distutils-test$ PATH= /usr/bin/python setup.py install --prefix /tmp/distutils-test
running install
running build
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-i686-2.5
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.5 -c foo.c -o build/temp.linux-i686-2.5/foo.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
exarkun@boson:~/distutils-test$ 

Is there some way to tell distutils that it's okay to not build an extension? If not, I think you should file a ticket in the Python tracker asking for this feature and we can use it once it's implemented.

comment:5 Changed 6 years ago by zooko

Okay, I opened a ticket for distutils:

http://bugs.python.org/issue4706

That ticket claims that distutils already handles the case that gcc exited with a non-zero return value, but not the case that gcc couldn't be found.

I also added this to the list of issue tickets on my klog.

I can't help thinking that there ought to be a way for Twisted to make this work better before they fix it in Python, they release a new version of Python with the fix, OLPC upgrades to that new version of Python, and Irby upgrades to that new version of OLPC. That chain of events might never actually happen, not least because I'm currently advising Irby to stick with Python 2.5 and not upgrade to newer versions of Python. ;-)

Just to throw out an idea, how about an argument to setup.py that says --please-no-c-modules? It's ugly in various ways, but at least Irby could deploy Twisted onto his OLPC that way.

Another idea would be to provide a subclass of the distutils build command which ignores that exception and converts it into a "normal failure" to build.

comment:6 Changed 6 years ago by exarkun

I'd like to see what happens to http://bugs.python.org/issue4706 before trying to come up with a solution of our own.

comment:7 Changed 5 years ago by exarkun

Perhaps the resolution to http://bugs.python.org/issue5583 (which issue4706 was closed as a duplicate of) gives us what is necessary here.

http://svn.python.org/view/python/trunk/Doc/distutils/setupscript.rst?r1=70910&r2=70909&pathrev=70910 is mostly the relevant part.

comment:8 Changed 5 years ago by zooko

I encountered this issue again recently. I've written some software for internal use at a corporation. They wanted to deploy it to their Windows XP server. I gave them a README.txt saying to run "python ./setup.py install" to install my software along with all of its dependencies, which are:

pyutil 1.4.1,
argparse 1.0,
lxml 2.2.2,
SQLAlchemy 0.5.5,
pyodbc 2.1.6,
setuptools 0.6c9,
txAMQP 0.2,
zbase32 1.1.1,
Twisted 8.2.0,
zope.interface 3.5.2,
setuptools 0.6c9

Two of the packages didn't install when they tried to deploy it on the Windows servers. One was txAMQP because it isn't currently reachable from PyPI (see https://bugs.launchpad.net/txamqp/+bug/413691 ). The other was Twisted, because they don't have a compiler on their Windows server. I had to help the Windows server admin download the .exe installer from the twistedmatrix.com web site. When he saw the name of it, he said "Are you sure this is safe?". I assured him that it was a standard Python tool.

So if I understand correctly, once Python 2.7 comes out, then the distutils that comes with that will have this flag with which Twisted can specify that its extensions are optional, and that failure to compile them should not stop the build.

What about for users of older Pythons such as 2.5? (That's the version of Python on this company's Windows server, and it is also the version that I use and support for other cases nowadays.) I saw that the simplejson package has a feature of attempting to compile its extension module and then falling back to pure Python if it fails:

http://simplejson.googlecode.com/svn/trunk/setup.py

I emulated this technique in the setup.py for one of my projects (zbase32) and it worked fine. Perhaps we should do likewise for Twisted?

comment:9 Changed 4 years ago by tarek

Distutils2 has an option to make extensions optional.

Until then you can write a custom build_ext command that handles the compilation errors, and use it in setup.py

Simplest form:

    from distutils.errors import CCompilerError, DistutilsError, CompileError
    from distutils.command.build_ext import build_ext as distutils_build_ext

    class build_ext(distutils_build_ext):

        def build_extension(self, ext):
            try:
                distutils_build_ext.build_extension(self, ext)
            except (CCompilerError, DistutilsError, CompileError), e:
                pass
   

comment:10 Changed 4 years ago by tarek

Oops, just read the simplejson example, it's pretty similar.

comment:11 Changed 4 years ago by zooko

This patch appears to fix it on my manual testing on Mac OS 10.4.

Changed 4 years ago by zooko

comment:12 Changed 4 years ago by zooko

As with many of my intended contributions to Twisted, I have trouble writing a test for this once, since the test probably needs to be run from a shell or a buildbot BuildStep or something rather than from trial.

I *guess* it might be possible to run trial and have a test case which sets up an environment in which it can invoke "python setup.py build" on a Twisted source tree and that no C compiler answers the call, by messing with the PYTHONPATH and the PATH and so on. Shall I try that? Advice would be welcome. I would like this patch to be accepted into Twisted since I'm currently helping yet another person install Twisted on a machine with no compiler...

comment:13 Changed 4 years ago by zooko

  • Cc zooko@… added

comment:14 Changed 4 years ago by exarkun

Would it be possible to have a unit test which fiddled with PATH and invoked the customized build_ext (as opposed to actually running Twisted's setup.py and installing all of actual-Twisted)?

Changed 4 years ago by soult

comment:15 Changed 4 years ago by soult

Hi there, I tried to install Twisted without a compiler on Windows 7 64bit with Zooko's patch. Using Twisted 10.0.0, I got the following error:

running build_ext
error: Unable to find vcvarsall.bat

With Zooko's help I was able to find out that distutils returned DistutilsPlatformError("Unable to find vcvarsall.bat"). After adding an except block for it, it worked like a charm (besides the warning about not using the c extension for speedup).

It would probably be best to make distutils return the correct error instead of a DistutilsPlatformError instead of using this hack.

The attached patch includes the previous patch plus my workaround.

comment:16 Changed 4 years ago by soult

  • Cc david+twisted@… added

comment:17 Changed 4 years ago by zooko

  • Owner changed from radix to zooko
  • Status changed from new to assigned

exarkun: I will try. soult: thanks for the patch.

comment:18 Changed 4 years ago by thijs

  • Cc thijs added
  • Keywords review added
  • Owner zooko deleted
  • Status changed from assigned to new

Putting the patch up for review.

comment:19 Changed 4 years ago by thijs

  • Description modified (diff)

comment:20 Changed 4 years ago by exarkun

  • Keywords review removed
  • Owner set to soult

This looks like a good start. If we could also add unit tests for the new error handling behavior, then I think we can probably apply the change.

comment:21 Changed 4 years ago by zooko

  • Owner changed from soult to zooko
  • Status changed from new to assigned

comment:22 Changed 4 years ago by davidsarah

Can't this be adequately tested by mocking build_ext.build_ext.build_extensions to raise a CCompilerError and a DistutilsPlatformError?

comment:23 Changed 4 years ago by davidsarah

  • Keywords build added

Related to #4612 (about building Twisted on Win7 64-bit with no Microsoft compiler).

comment:24 Changed 4 years ago by exarkun

Mocking might be the sanest testing solution. We want to be confident the test actually exercises the code that will be invoked when people run this on the commandline, of course. If there's an API we can invoke (distutils.core.setup perhaps?) which lets us have that confidence, excellent. The other option is to run a child process, which isn't as shiny, but at least obviously exercises the right code.

comment:25 Changed 4 years ago by glyph

#4720 looks like a proposal for a solution to this problem. Is it a duplicate?

comment:26 Changed 4 years ago by davidsarah

  • Cc davidsarah added

It looks like a duplicate to me, but it also points out that it would be useful to have a way to require particular extensions:

There is a problem with exit codes, though: someone may want the build to stop if a module they need fails to build. I (hastily) propose some kind of --needcexts=x,y,z option, and if any of those modules fail to build, exit with a non-0 code.

Rather than a command-line argument, this could be expressed as an optional setuptools/distutils feature requirement, e.g. "twisted[iocpreactor]" (see #4438 for how to do that).

I wouldn't want this to delay fixing the main issue in this ticket, though.

comment:27 Changed 4 years ago by ivank

  • Cc ivank-twisted-bugs@… added

I applied patch2.txt. setup.py failed to compile portmap, emitted a warning, then didn't try to compile any of the other C extension modules (installation did succeed though). Is this the intended behavior?

comment:28 Changed 4 years ago by fijal

  • Cc fijal added

The same problem appears when trying to install twisted on top of pypy in virtualenv (multiple people came complaining to #pypy so far). It's not the lack of gcc, but other bits in compilation not working (but the effect is the same and the same patch should fix it)

comment:29 Changed 4 years ago by techtonik

  • Cc techtonik@… added

comment:30 Changed 4 years ago by techtonik

How about making an explicit --no-speedups option for the start? Explicit behavior will allow to fail early if compiler for speedups is not detected or installed.

Changed 4 years ago by techtonik

comment:31 Changed 4 years ago by techtonik

  • Keywords review added
  • Owner zooko deleted
  • Status changed from assigned to new

Added global --no-speedups option. There is a limitation though - this works:

setup.py --no-speedups bdist

This does't:

setup.py bdist --no-speedups

I couldn't find a way to patch distutils.cmd.Command.__getattr__() to pass global option through this check.

Changed 4 years ago by techtonik

no leftovers

Changed 4 years ago by techtonik

(removed unused build import)

comment:32 follow-up: Changed 4 years ago by exarkun

  • Keywords review removed
  • Owner set to zooko

I'd rather not add non-standard options to setup.py. This also doesn't help people using easy_install, since they can't pass options to bdist.

comment:33 Changed 4 years ago by techtonik

There is a conflicting story for making extension compilation implicitly unnecessary: "I do not want to install Twisted without speedups"

We need to come upon a consensus which scenario is primary.

comment:34 in reply to: ↑ 32 Changed 4 years ago by glyph

Replying to exarkun:

I'd rather not add non-standard options to setup.py. This also doesn't help people using easy_install, since they can't pass options to bdist.

We could address the easy_install problem by having someone build binary eggs for each supported platform.

comment:35 Changed 4 years ago by exarkun

We could address the easy_install problem by having someone build binary eggs for each supported platform.

But OLPC isn't a supported platform. :)

comment:36 Changed 3 years ago by zooko

I see that #5158 has been merged and released in Twisted 11.1.0. That's good! The patch that fixes #5158 does so by checking whether the platform is (probably) CPython in addition to other checks, such as whether the sys/epoll.h header file exists, in order to decide whether to attempt to build certain extensions.

This is what I call the "check if it will work and then do it" approach. An alternative approach that is more general would be the "try it and don't worry if it didn't work" approach.

That approach would hopefully fix both this ticket (build Twisted on a platform with no C compiler) and #4612 (build Twisted on 64-bit Windows with mingw), as well as not regressing #5158 (build Twisted on PyPy). (As fijal mentioned in comment:28.)

My patch2.txt for this ticket implements that approach (copied from simplejson's setup.py).

I'd like to try to get patch2.txt merged, again. Here's what I think is needed. Please let me know if this is correct:

  1. First of all, patch2.txt doesn't have unit tests. At the time, I hadn't yet grokked mockery in its fullness, and without mockery it was complicated to write tests of build logic, where the test code would have to be buildsteps executed by the buildbot instead of unit test methods executes by trial) (see comments 20, 22, 24). #5158 has demonstrated that mockery is indeed a good way to test this sort of thing. I think we can do even better than the unit tests in #5158, by having unit tests which actually invoke build_ext_twisted.
  1. As ivank reported in comment:27, the current patch2.txt skips building of all extensions after building of an extension fails. This should probably be changed to attempting to build each extension separately, in case some of them fail but others succeed.
  1. Please take a minute to think whether there are problems with the "try it and don't worry if it didn't work" approach. That is: could it be wrong or damaging to attempt to build on a platform where the build will not succeed? Possible problems: 1. it takes up resources such as time or RAM unnecessarily. I don't think this is enough of a problem to worry about. 2. it could have some side-effect such as creating a directory of file during the attempt to build, the presence of which causes some other problem later. I'm not aware of any such problematic side-effects from attempting to build (I'm familiar with Linux, cygwin, Windows-MSVC, Windows-mingw), but if you are please speak up. In any case, remember that there are also potential problems from the "check if it will work and then do it" approach, namely that the "check" step may falsely think that it will work when it won't or think that it won't work when it will.

Okay, if someone will give me a word of encouragement, then I'll proceed to write code for 1 and 2 above (mockery-oriented unit tests and independent attempts per extension).

comment:37 follow-up: Changed 3 years ago by exarkun

At the time, I hadn't yet grokked mockery in its fullness, and without mockery it was complicated to write tests of build logic, where the test code would have to be buildsteps executed by the buildbot instead of unit test methods executes by trial)

Or, the tests could be unit test methods executed by trial (as I mentioned in comment 24). Mocking should be a latter tool to reach for in testing, not an earlier one. What are the obstacles to writing a unit test that actually exercises the code?

Also, another drawback to "try it and don't worry if it didn't work" (aka "easier to ask forgiveness than permission" aka EAFP) is that the build might succeed and produce an extension that is wrong when used (wrong results, crashes, etc).

comment:38 in reply to: ↑ 37 Changed 3 years ago by zooko

Replying to exarkun:

What are the obstacles to writing a unit test that actually exercises the code?

The code I'd like to see exercised is build_ext_twisted. The questions that I would like the test to answer about this code are things like "Does it proceed to build extension B even after attempting to build extension A incurred an error?".

But, if when running the test we actually invoke build_ext_twisted and it actually invokes build_ext.build_ext.build_extensions, then a side-effect of running the unit tests will be that the extensions get rebuilt! That sounds potentially troublesome, and we don't need to test that build_ext.build_ext.build_extensions (that's distutils's responsibility, not ours), so I propose to patch out build_ext.build_ext.build_extensions and replace it with a mock object whose job it is to make sure that we pass the correct arguments to build_ext.build_ext.build_extensions (our responsibility, not distutils's).

What do you think?

Also, another drawback to "try it and don't worry if it didn't work" (aka "easier to ask forgiveness than permission" aka EAFP) is that the build might succeed and produce an extension that is wrong when used (wrong results, crashes, etc).

That's a good concern to keep in mind. Do you think that we can rely on the build scripts of our extensions on our platforms to not do that? Or should we abandon this whole approach? Note that the "check if it will work and then do it" approach could also encounter this problem, if the check step decides that it will work when it won't.

comment:39 Changed 3 years ago by exarkun

so I propose to patch out build_ext.build_ext.build_extensions and replace it with a mock object whose job it is to make sure that we pass the correct arguments to build_ext.build_ext.build_extensions (our responsibility, not distutils's).
What do you think?

I think the difficulty with this is that I don't think any parts of distutils are considered public or stable. A mock runs the risk of becoming desynchronized with the real implementation. (Of course, since Python is effectively unmaintained - or at least in feature freeze - at this point, perhaps this concern should be given less weight.)

However, I do think your concern about actually building Twisted's extension modules is misplaced. The unit test should not try to build Twisted's extension modules. It should try to build some test-provided dummy extensions, at least one of which can be guaranteed to fail to build (perhaps by having a syntax error or including a non-existent header file). Twisted's own extension modules are data for the code in question. Lots of tests exercise code with made up data.

That's a good concern to keep in mind. Do you think that we can rely on the build scripts of our extensions on our platforms to not do that? Or should we abandon this whole approach? Note that the "check if it will work and then do it" approach could also encounter this problem, if the check step decides that it will work when it won't.

A combination of the two makes sense to me. We can skip configurations that we know are invalid so that we don't have to worry about a build inadvertently succeeding. We can also have a safety net, ignoring build failures, so that even things we think should build and work don't prevent build/install from completing if they can't be built.

I think the question of whether it is actually most useful to quietly proceed after build failures hasn't been completely explored, and at some point we may really want build failures to interrupt the normal build/installation process. However I'm happy, for now, to see them not do that, since the current behavior seems to be more troublesome.

comment:40 Changed 21 months ago by Scramblejams

Just wanted to say this is still an issue -- I needed to apply Zooko's patch.txt in order to do a --user install of Twisted 12.3.0 on RHEL 6.0 which didn't have the Python dev libraries installed. Thanks Zooko!

comment:41 Changed 11 months ago by exarkun

#6853 was a duplicate of this.

#6831 may be related in some interesting way as well.

Note: See TracTickets for help on using tickets.