make twisted play nicely with easy_install

[icepick]

I want to play with divmod.org's Axiom, but I see I need a new version of
Twisted, Twisted Web, and also I need Epsilon.  Since my apt sources don't have
packages, I would like to use easy_install
http://peak.telecommunity.com/DevCenter/EasyInstall
The ideal way this should work is:
 % sudo easy_install Axiom
and then play with Axiom.  With easy_install this is possible, but there are
some minor things needed changed in Twisted to get this working (as well as
minor things to be done in Axiom's setup.py, but this is the Twisted issue
tracker, so I'll stick with Twisted changes)
1) Need Twisted Sumo - Why?  Because if you install Twisted and Twisted Web as
two seprate packages they end up in:
/usr/local/lib/python2.3/site-packages/Twisted-2.1.0-py2.3.egg/twisted
/usr/local/lib/python2.3/site-packages/Twisted_Web-0.5.0-py2.3.egg/twisted/web
and if you try python -c "import twisted.web" it will tell you "ImportError: No
module named web".
2) Fix t.p.dist.  In this file there are these lines:
    if 'detectExtensions' in kw:
        if 'ext_modules' not in kw:
            kw['ext_modules'] = [True] # distutils is so lame
"[True]" makes easy_install barf.  Why is it needed?  What is this mysterious
comment about distutils being lame?  I replaced "[True]" with "[]" and
everything worked, so it would help if that comment was more illumating.
3) URL of package on project page.
If you just try
 % sudo easy_install Twisted
easy_install hits the cheese shop, then http://twistedmatrix.com/projects/core/
and looks for a likely package URL.  I'm guessing if the Sumo package is named
TwistedSumo-xxx.tar.bz2 easy_install will pick the non-sumo package, thus
causing the problem I mentioned in 1.  humm... ideas:
 a) patch python so twisted and twisted.web can exist and work in diffrent path
entries
 b) go with the flow and rename twisted.web to twistedweb or
theonlywebframeworkyoureallyneed, or some such, rather than having a separately
distributed package put it's files into another package's folder.
 c) patch easy_install to understand when this happens and use symlinks or
something to get around it.

[radix]

READY FOR REVIEW in easy_install-1286

[therve]

Cosmetics:

• trailing whitespaces in t.p.dist
• unused import sysconfig in t.p.dist
• trailing whitespaces in test_dist
• tearDown method uses """string""" format
• unused imports in twisted/topfiles/setup.py
• copyright statements should be added to t.p.dist, test_dist, twisted/runner/topfiles/setup.py, and updated to 2007 for twisted/topfiles/setup.py
• dist.Extension has a bit disturbing name, that does make it clear it's different from distutils Extension. I'm not very inspired, maybe ConditionalExtension ?

OK now I'll test the functionality :).

[therve]

I may miss something, but it seems it doesn't work with the 'sumo' package. I could install each subprojects, but it's not compatible (see first comment).

So, do I miss something?

[radix]

Sorry, yeah, it doesn't work to easy_install the full Twisted release. That'll be much more work. This makes it possible to easy_install the individual subprojects.

If you want to test it, you can do ./admin/release-twisted --commands=exportTemp,makeBallAll

(if you want to rerun that command, you will either have to get rid of Twisted.exp or don't specify exportTemp again)

and then try easy_installing some of the tarballs created. (no dependency tracking is done yet, but I consider that a lack of a feature instead of a bug for now).

[therve]

So the problem is with the top-level setup, and I may have a solution. The problem is our hand-made setup.py that allows to install sub-projects. But, we can probably have a simple setup.py that just installs all of Twisted, right ?

What I'd do is import all the subprojects setup.pys, to get extensions (there are no extensions outside of core ?) and other stuff, and just run one setup for all of this. I've quickly tried it, and it seems to work without problem. It installs everything of Twisted... but we may find a way around, until we understand easy_install.

Of course, we can keep current setup.py around under another name for people who cares.

[therve]

To workaround the namespace problem, we must use setuptools: http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages.

zope seems to have a nice way to workaround the absence of setuptools. See the packaging of zope.interface (it seems their setup.py is generated).

[therve]

I've made a simple, stupid setup.py that install everything in a egg (21M of pure happiness). It detects the extensions, you should have to add __init__.py in twisted/topfiles. I consider that subprojects don't have extensions, which is true for now.

[antoine]

Just a small thing: your setup script munges sys.path after it has imported copyright and dist. Is it deliberate?

[exarkun]

This seems to be up for review.

[radix]

I guess things have changed in trunk since this setup.py was written, but it doesn't work any more. It doesn't seem to be a trivial fix, either.

[therve]

(In [21261]) Create the new setup, yet another workaround for building extensions.

Refs #1286

[therve]

I didn't remember the first version, so I tried something else, but this time in a branch: easy_install-1286-2.

THe only real thing to notice is that it doesn't detect extensions anymore, but doesn't fail when with the compilation.

[zooko]

I guess this would also solve #256.

What's the next step for this ticket? Does therve need to do something more before it is ready to review?

[therve]

It is ready to review, see the keyword.

[exarkun]

What user-visible functionality should I expect from the easy_install-1286-2 branch?

[exarkun]

Uggg. As an unrelated thing, the setup.py add/delete causes svn 1.3.2 to corrupt the working copy when merging this branch. Perhaps that file could be M instead of D/A?

[therve]

Replying to exarkun:

What user-visible functionality should I expect from the easy_install-1286-2 branch?

The twisted setup.py is now compatible with easy_install. For example, in a checkout of the branch, you can do 'easy_install .', and that should work.

[exarkun]

Pretend I've never used easy_install. :) Is easy_install . the only feature I should look for when reviewing this?

[zooko]

The biggest user-visible change that I'm interested in is that it makes Twisted's dependencies on other packages machine-readable. Wait, that isn't user-visible.

Okay, the biggest user-visible change that I'm interested in is that I can tell the tahoe setup.py that tahoe depends on Twisted, and when someone invokes tahoe's setup.py, then tahoe's setup.py will parse Twisted's dependencies to learn that Twisted depends on zope.interface, and then tahoe's setup.py will make sure that zope.interface is installed before installing twisted.

So the user-visible difference from my perspective is like this:

1. Current situation -- tahoe documentation instructs reader to install Twisted and zope.interface before proceeding.

2. Twisted packaged with distutils -- user has to manually install zope.interface (because twisted doesn't declare its dependency on zope.interface in a machine-readable way), but user doesn't have to manually install Twisted -- it gets installed automatically because Tahoe depends on it.

3. Twisted packaged with setuptools -- Twisted and zope.interface get installed automatically because Tahoe depends on Twisted and Twisted depends on zope.interface.

A way to simulate this without actually running Tahoe's setup.py is to install easy_install and execute "easy_install ." in the twisted dir that contains twisted's setup.py. (I think.)

There are other issues too, but this is the motivating one, for me.

[zooko]

Note:

It should be possible to use setuptools as your packaging tool and continue to produce packages of the traditional form: .deb's, Windows Installer Thingies, source tarballs, etc.. Setuptools supports all of these (just like distutils does -- perhaps it actually works merely by delegating to distutils). So nobody should think that by gaining the ability to produce eggs, it will become harder to produce other packages. Indeed, this patch will probably make it easier to produce those other packages, by invocations like "./setup.py sdist" and its brethren.

I guess this suggests a couple of more "user visible" things to look at:

Can you run "./setup.py sdist" and get a source tarball that has all the right files in it?

Can you run "./setup.py bdist_wininst" and get some kind of Windows Installer Thingie? I don't know much about this part -- it might not work at all.

[zooko]

Here's one drawback to using setuptools. It changes the meaning of PYTHONPATH from "Here is a list of places to look for a module such that you will use the first module of that name in this list." to "Here is a set of places to look for modules such that you will use whichever module that appears in this set that setuptools chooses (based on the module's version number).". This is a backwards-incompatible and "silently confusing" change in semantics.

http://www.rittau.org/blog/20070726-02

The suggested fix from the setuptools folks is to use "./setup.py develop", which installs the package into your local system with package contents being symlinks back into your current directory.

I sincerely hope that this issue is either work-aroundable or else is considered to be a drawback which is worth accepting in return for the advantages.

[therve]

FWIW, I don't think we plan using setuptools. The goal of this ticket (I think) is just to make the setup.py compatible with easy_install. easy_install doesn't force you to use setuptools at all.

[zooko]

Okay, so given that the purpose of this ticket is merely: "make Twisted play nice with easy_install", then all we require is for Twisted to declare its dependencies and version number in a machine-readable and standard way. As far as I know, the easiest way to do that is to use setuptools instead of distutils for packaging, and pass "install_requires=zope.interface?" as one of the arguments to setup().

If Twisted declares its dependencies and its own version number in the standard way then my tahoe project will be able to automatically satisfy its dependency on Twisted and Twisted's dependency on other projects (zope.interface) without user intervention.

There are of course other ways to accomplish this than invoking setuptools's "setup()". One could construct the appropriate twisted.egg-info/requires.txt files, for example.

[pje]

Actually, the simplest way to make it "play nice" for the dependency would be to check if setuptools is already in sys.modules. If so, then setup.py will know it is being run under easy_install, and can declare additional options like install_requires.

Personally, I'd be happy to be able to get Twisted to install via easy_install, even without the zope.interface dependency, because then at least people could manually include a zope.interface dependency in their projects as a workaround.

As far as the review/testing goes, I would say the objective is that you should be able to run the distutils "sdist" command to make a source distribution that can be passed on the command line to easy_install.

That is, if you run "setup.py sdist" followed by "easy_install dist/Twisted-whatever.tar.gz", you should get a working Twisted installation on sys.path, with all the extensions, data files, or whatever else included.

Ideally, the aforementioned sdist would also be registered and uploaded to the Cheeseshop, but a "download URL" on the Cheeseshop pointing to a page with *direct* download links would also work.

By the way, easy_install also supports installing from bdist_wininst .exe files, so that's a distribution option as well. However, it can't install arbitrary .exe's, they *have* to be built with bdist_wininst (which includes some extra data in the .exe that easy_install depends on).

Last, but not least, the only other thing that hampered easy_install is inconsistency in package names/distribution file names. That is, easy_install sees a distro like "Twisted_NoDocs-SomeVersion?.tar.gz" as a package called "Twisted_NoDocs" of version "SomeVersion?", and therefore ignores it as an installation target when someone asks for "Twisted" to be installed.

In other words, when it comes to distribution formats, easy_install really only supports files generated by the distutils (sdist and bdist_wininst) and which retain their distutils-defined filenames. (It also does SVN checkouts and eggs, but those are sort of beside the point at the moment.)

[zooko]

Okay, when I run "./setup.py sdist" on current trunk, it fails. c.f. #2886.

When I run "./setup.py sdist" in the branch, it builds a tarball named "Twisted-2.5.0+r21763". The names of the files in that tarball are attached to this comment as a text file attachment. When I run "easy_install ./Twisted-2.5.0+r21763", then it gives me this failure message:

  File "./twisted/__init__.py", line 22, in <module>
ImportError: you need zope.interface installed (http://zope.org/Products/ZopeInterface/

If I install zope.interface and try again, then it works:

Processing dependencies for Twisted==2.5.0
Finished processing dependencies for Twisted==2.5.

This is on Mac OS 10.4/Intel. It built some extensions -- c_urlarg, cfsupport -- and it didn't let the failure to build others -- epoll, iocp, -- stop it.

[zooko]

listing of the contents of the twisted tarball built by "./setup.py sdist"

[zooko]

patch to make twisted's setup.py declare its dependency on zope.interface, but only if it is being evaluated by easy_install

[zooko]

patch to make twisted's setup.py declare its dependency on zope.interface, but only if it is being evaluated by easy_install

[zooko]

As per pje's suggestion in this ticket, I wrote the attached patch, which declares Twisted's dependency on zope.interface, but only if setup.py is being evaluated by easy_install. With this patch, then the behavior under normal conditions (not-being-evaluated-by-easy_install) is unchanged, but when I run easy_install ./Twisted-2.5.0+r21763 then it detects the requirement of zope.interface and easy_install automatically satisfies that requirement.

[therve]

(In [21808]) Apply zooko's patch.

Refs #1286

[jml]

Here's my review as a non-expert in easy_install & friends.

twisted/python/dist.py

• Need spaces between operators (see PEP 8) in build_ext_no_fail

setup.py

• Needs a module docstring.
• Needs a docstring for main()
• Duplicates significant amounts of data from twisted.topfiles.setup. It would seem more sensible to copy the dict and mutate the values, or at least refer to values in the topfiles setup_args dict.

I also can't find out where setupdist.py is used. grep doesn't help me.

I also think this branch needs RM approval before landing.

[zooko]

therve: why did you create a setup_args dict in setup.py instead of using the one that was in twisted/topfiles/setup.py? Should one version replace the other?

[zooko]

The answer to "what is setupdist.py for" is that it is just vestigial since therve mv'ed it aside when he started.

Here is a patch which adds spaces between operators, adds a module docstring and a main() docstring, and rm's setupdist.py.

[zooko]

Okay here's a patch which fixes the last of the issues that jml noted:

fix problems noted by jml in http://twistedmatrix.com/trac/ticket/1286#comment:37

• spaces between operators
• module docstring
• main() docstring
• import setup_args metadata values from twisted.topfiles.setup instead of duplicating them in source code
• remove old setupdist.py

[exarkun]

(In [21912]) Apply zooko's patch

Refs #1286

[exarkun]

(In [21913]) Branching to 'easy_install-1286-3'

[exarkun]

Failing test in the branch: twisted.test.test_doc.DocCoverage.testPackages

The "docstring" on the if-suite in the top-level setup.py looks like it really belongs on the main function in that file.

Lots and lots of files don't end up in the sdist tarball which results from setup.py sdist, which is the intended use of this stuff, as far as I can tell. That means the resulting install can't possibly be completely correct, since it won't start with the necessary files. Lots and lots of tests fail when run against the result of unpacking the tgz created by the sdist subcommand.

I don't really understand what the new setup.py is doing, so I haven't really reviewed it, I've only looked at the resulting behavior, which doesn't seem to be what is desired.

[therve]

I fixed some stuff in the branch, but I have to look further.

[zooko]

If you want my help with this branch, you'll have to answer my questions that I posted in this ticket.

In a related story, I might actually go ahead and spend time to implement some automated tests of the Twisted install process, as per #2308 . The basic idea would be to use Nexenta Zones to make a virtual system into which I can install and then rollback the whole system. If I do something like this, then how should the tests be written? My current idea is that they should be in the form of a bash script that initializes the zone, attempts to install Twisted within that virtual operating system, runs the Twisted unit tests therein, and exits with code 0 if the twisted unit tests passed.

[zooko]

Oh, looking back over the ticket I haven't posted the questions that I need answered.

Basically: what is your strategy? How should the current Twisted setup scripts be changed to enable Twisted to be easy_install'able? As far as I understand, two possible strategies stick out:

1. Use setuptools instead of distutils to package Twisted. This strategy is attended by a lot of emotion from some people that I don't quite understand, but I don't think the emotion is that big a deal, compared to the fact that this strategy has technical consequences that can be automatically evaluated as documented in #2308 . For what it is worth, I am maintaining a patch for my own use which does this. The patch is near-trivial, and the result is conveniently usable for my purposes.

2. Detect at build time whether pkg_resources is imported. If it is, then we can safely add an install_requires=['zope.interface'] item to the dict which is passed to setup().

Actually, this raises another point about which I am confused: except for the issue about declaring dependencies (on zope.interface) in a programmatic way, aren't all packages which are built with a modern version of distutils automatically easy_install'able? What do the twisted setup scripts do which prevents this from working?

[radix]

Zooko: To answer your questions about automating the tests, I think the only requirement I care about is that it be a buildslave. Making it debuggable is another concern; if something goes wrong, can we duplicate this setup locally?

To answer your other question, the reason that the current setup.py is not easy_installable is that not only does it not use setuptools, it also doesn't use distutils at all. We need to change the toplevel setup.py to be a regular distutils-based script, instead of the hack it is which runs a bunch of other setup.py files.

The stuff you said about pkg_resources sounds fine.

[zooko]

patch to build twisted with setuptools instead of distutils

[zooko]

I just attached a patch that I've been using for a while to build twisted using setuptools. This causes twisted to be easy_install'able.

[radix]

(In [22977]) Branching to 'easy_install-1286-4'

[radix]

The referenced branch allows Twisted to be easy_installed *and* actually builds all the extension modules. It still needs some basic work, like removing some duplication and unit testing some of the simple changes to dist.py, but really I need to set up a buildslave that runs the setup.py and then runs the tests inside of them. The twistedmatrix.com buildbot configuration is overwhelming to me at this point, so it would be great if someone who's familiar with it would pair with me on that. I think exarkun, therve, and zooko would be the best to help me with this, in that order, based on my understanding of THEIR understanding of our configuration :)

[therve]

OK I think we should create a buildstep that:

• create a distribution tarball for Twisted (the big one without web2 I guess)
• call easy_install tarball.tar.bz2 --install-dir=something
• run the test suite in something. We can use the epoll reactor to be sure that extensions are built

I'm not sure how hard it would be, though.

I've also tried the branch and it does the right thing for me.

[therve]

(In [22992]) Branching to 'easy-install-1286-4'

[radix]

Ok, this is working now, and with the help of therve, exarkun, zooko, and idnar we now have a buildslave: "debian-easy-py2.5-epoll".

There's one thing I'm curious about, and maybe zooko can help. The 'name' field we specify in the setup.py files is not a python package name, as seems to be the convention for setuptools-based packages. Is this necessary? If we switch to package names, we have a problem: both the Twisted core distribution as well as the main Twisted distribution would have to be called "twisted". Is it ok if you have to depend on "Twisted >= 8.0" instead of "twisted >= 8.0"? That way if somehow we ever get our subprojects working with setuptools, one could depend on "Twisted Core >= 8.0 and Twisted Web >= 8.0", or whatever.

I'm not putting this up for review *quite* yet because there's one failing test on the easy install slave related to process.alias.out, so I'm going to investigate that.

[radix]

seems to be because easy_install isn't maintaining privs on data files. process.alias.sh isn't executable after easy_install, and the test relies on executing it. Any opinions on what I should do, anyone? I checked and this looks like the only case where the test suite is executing a data file or otherwise relying on perms of data files - I could just make the test mark it executable. I could also try to figure out how to make easy_install maintain privs, but obviously that will be harder.

[radix]

chmodding is bad because then the test won't work in an installed environment, unless I trap and ignore the exception. bleh. good night.

[therve]

I've looked at the problem and I don't think that expecting rights will remain the same after install is a good idea. So I think we should fix that test: I have a local fix, which create a temporary file in _trial_temp, set it executable, and use it in the test. If there is an agreement about this being the good solution, I'll create a ticket and push the branch.

[radix]

That sounds fine, can you please make a branch

[radix]

Ok, therve did a branch at #3110 and it's been merged and I merged forward this branch. debian-easy-py2.5-epoll is now completely green. Please review.

[therve]

• pyflakes errors:

  twisted/python/dist.py:11: 'CCompilerError' imported but unused
  twisted/runner/topfiles/setup.py:5: redefinition of unused 'Extension' from line 2
  

• trailing whitespaces in test_dist
• test_win32Definition should use self.patch

Look a bit under windows.

[therve]

OK, +1 for me.

[radix]

(In [23006]) Merge easy_install-1286-5

Author: radix Reviewer: therve Fixes #1286

The top-level setup.py file has been rewritten as a normal distutils script. It also added optional support for setuptools, so that it's possible to easy_install Twisted and even have it automatically download zope.interface if necessary. twisted.python.dist has changed so that it now accepts a list of Extension objects which know how to check if they should be built, rather than a function that dynamically creates a list of buildable Extensions. This made sharing between subproject and main Twisted setup.py files easier.

[radix]

(In [23007]) Revert r23006: Buildbot failures related to setuptools

It seems to be related to old versions of setuptools not accepting our setup.py file:

error in Twisted setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers

Reopens #1286

[radix]

Up for review again.

I changed it to skip dependency specification if we're using a crappy version of setuptools. seems to work on cube.

[radix]

exarkun has +1ed this on IRC.

[radix]

(In [23010]) Merge easy_install-1286-5

Author: radix Reviewer: therve, exarkun Fixes #1286

The top-level setup.py file has been rewritten as a normal distutils script. It also added optional support for setuptools, so that it's possible to easy_install Twisted and even have it automatically download zope.interface if necessary. twisted.python.dist has changed so that it now accepts a list of Extension objects which know how to check if they should be built, rather than a function that dynamically creates a list of buildable Extensions. This made sharing between subproject and main Twisted setup.py files easier.