build executables for trial, twistd, manhole, etc. on Windows (if setuptools is installed)

[zooko]

I wanted to run trial on Windows. I started with the Twisted 8.2.0 source distribution, and ran python ./setup.py install, which seemed to succeed and even printed out encouraging messages about installing trial and twistd and so forth. Then I ran trial --version, but it said:

'trial' is not recognized as an internal or external command, operable program or batch file.

I then installed setuptools on this Windows box and tried again, with the same result.

Then I applied the following patch to Twisted's setup.py and installed again. (I now have setuptools installed on this Windows system, so the "setuptools branch" of Twisted's setup.py is used -- this is the same thing that happens if someone runs easy_install Twisted.) This time when I run trial it executes correctly! Hooray!

>trial --version
Twisted version: 8.2.0

Patch:

--- setup.py.orig       2008-09-06 14:55:38.000000000 -0600
+++ setup.py    2009-06-11 12:53:23.453125000 -0600
@@ -89,6 +89,25 @@
             setup_args['install_requires'] = requirements
         setup_args['include_package_data'] = True
         setup_args['zip_safe'] = False
+        setup_args['entry_points'] = {
+            'console_scripts': [
+                'manhole = twisted.scripts.manhole:run',
+                'mktap = twisted.scripts.mktap:run',
+                'pyhtmlizer = twisted.scripts.htmlizer:run',
+                'tap2deb = twisted.scripts.tap2deb:run',
+                'tap2rpm = twisted.scripts.tap2rpm:run',
+                'tapconvert = twisted.scripts.tapconvert:run',
+                'trial = twisted.scripts.trial:run',
+                'twistd = twisted.scripts.twistd:run',
+                'mailmail = twisted.mail.scripts.mailmail:run',
+                'lore = twisted.lore.scripts.lore:run',
+                'cftp = twisted.conch.scripts.cftp:run',
+                'ckeygen = twisted.conch.scripts.ckeygen:run',
+                'conch = twisted.conch.scripts.conch:run',
+                'tkconch = twisted.conch.scripts.tkconch:run',
+                'im = twisted.words.scripts.im:run',
+                ]
+            }
     setup(**setup_args)

[exarkun]

Use the Twisted command shell and trial, twistd, etc will work.

[zooko]

Ah, I don't know what the "Twisted command shell" is and I would really rather use cmd.exe or bash, as those are tools that I already know and understand. Also some of my invocations of trial are actually being done by execve(), so would I need to change my code which uses trial to use the Twisted command shell? And would the code for doing that have to vary by platform? Here's an example of code that I was just messing with which executes an executable named trial (or equivalently trial.exe) and expects it to work:

 http://allmydata.org/trac/tahoe/browser/misc/run-with-pythonpath.py

Also, please consider the other poor benighted other users out there who may install Twisted (with python ./setup.py install or with easy_install Twisted) and who would benefit from trial and twistd executbles then being installed.

[exarkun]

The Twisted command shell is cmd.exe. The Twisted Windows installer adds it to the start menu. If you run run-with-pythonpath.py in the Twisted command shell, it will find trial. Probably. The rest of the PYTHONPATH manipulation that it does, I dunno about.

[zooko]

Personally, I prefer not to run a Windows installer and I prefer not to use a special shell just to use Twisted commands. I would prefer for Twisted to be installed with python ./setup.py install or easy_install Twisted, like the other Python packages that I use, and for its executables to be executables in the operating system, like other software that I use.

The patch that I posted will not break things for those other people who like to use Twisted-specific tools (I hope).

[glyph]

I wouldn't mind seeing this happen, but let's not burden radix any more.

[glyph]

There are a vast number of subtle and frustrating issues at work here, but I'll try to give a brief summary of my own perspective.

Right now, the expectation is that users who use Windows will get the Windows installer, run it, and get a nice "twisted command shell" start menu icon that they can use to run Twisted programs. This works, as far as it goes. However, it's not a scalable approach; you can't have a Twisted command shell and a Zope command shell and a Divmod command shell (etc) which each do their own little custom setup dance, especially if you intend to develop applications using more than one of these systems. We could say that the problem here is a documentation problem, that it's not clear enough that we want users to use the Windows installer, but even if they all did they're not going to have a Twisted command prompt.

On UNIX-like platforms, the current 'setup.py install ...' will produce several scripts in bin/. If your PATH is set to point to that directory, these act like "executables" and do basically the right thing. This is all because we pass 'scripts=' to setup(). Distutils on UNIX does the right thing.

Distutils on Windows, according to zooko at least (and I don't believe he's in the minority here) does something else that isn't as useful by default. It creates a "scripts" directory, and puts ".py" files into it ("twistd.py", "mktap.py", etc). On linux the expectation is that the user knows how to set their PATH. On Windows, the expectation is a little more complex; it's that the user will have a file association for .py files that runs "python.exe" (you can  read about exactly what that means here, but the general idea is that "they've installed some version of Python"), they will have set their PATH environment variable to include the appropriate Scripts directory, and they will have set their PATHEXT to include the ".py" extension.

That is what users should be doing, today, if they want to install Twisted via setup.py and use a regular cmd.exe prompt on Windows. Zooko, maybe that tidbit will help you.

However, this technique is both more work and less useful than the UNIX-y equivalent. Two notable problems are that (A) what program runs .py files is stored in the registry, not in the environment, so you have to molest operating-system-wide mutable state in order to change what your shell is doing, and (B) only one program may be associated with .py files for a given operating system install (that's right, HKEY_LOCAL_COMPUTER, not HKEY_LOCAL_USER); so you lose the feature where you can install Twisted with python 2.5 and again with python 2.6 in a different directory and select which one you want just by setting your PATH.

setuptools noticed these deficiencies and devised a completely different way of setting up command-line tools on Windows. I'm not 100% clear on the details, but the gist is that you get a .exe file for every mainpoint you register. I assume that the .exe points to (or maybe, blech, bundles in, but I hope not) the appropriate version of python.

Given that, for a certain audience, this metadata provides a superior experience, I am not opposed to providing entrypoints and thereby providing this user experience on Windows via easy_install and friends. However, I am opposed to the patch as such, because:

1. No tests! Write some dang tests!
2. I believe this will cause the main setup.py to specify both scripts and entrypoints, which may cause weird behavior on other platforms.
3. This is a modification of the main setup.py with no corresponding modification of each subproject's setup.py. The meat of this change belongs somewhere else, probably in twisted.python.dist.
4. I strenuously object to specifying the metadata of which scripts Twisted has in yet another way which must be maintained in parallel. Right now, the crud in trunk/bin/ is authoritative. I am OK with leaving a few generated files in SVN for convenience's sake, but we have to have a clear-cut authoritative source for where those files get generated from. Also, setuptools entrypoints can't be that authoritative source: Twisted must remain installable without setuptools for the forseeable future.
5. I'd like to see some discussion of what the ramifications are for a non-Windows user of adding this metadata. Does anyone care? If not, should it be platform-specific?

[glyph]

adding some ccz

[zooko]

Your summary is right, Glyph. Another limitation of the PATHEXT approach is that you can't run the scripts with execve(), right?

Yes, the .exe's that setuptools generates just point to Python rather than bundling it in.

On other platforms, a script is generated instead of a .exe. Here is the /usr/local/bin/trial generated on Linux:

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'Twisted==8.2.0-r26987','console_scripts','trial'
__requires__ = 'Twisted==8.2.0-r26987'
import sys
from pkg_resources import load_entry_point

sys.exit(
   load_entry_point('Twisted==8.2.0-r26987', 'console_scripts', 'trial')()
)

load_entry_point() looks in the metadata (i.e. the text file named entry_points.txt in the Twisted.egg-info directory) where it finds:

[console_scripts]
tapconvert = twisted.scripts.tapconvert:run
conch = twisted.conch.scripts.conch:run
tap2rpm = twisted.scripts.tap2rpm:run
mktap = twisted.scripts.mktap:run
tap2deb = twisted.scripts.tap2deb:run
pyhtmlizer = twisted.scripts.htmlizer:run
ckeygen = twisted.conch.scripts.ckeygen:run
trial = twisted.scripts.trial:run
im = twisted.words.scripts.im:run
manhole = twisted.scripts.manhole:run
mailmail = twisted.mail.scripts.mailmail:run
tkconch = twisted.conch.scripts.tkconch:run
lore = twisted.lore.scripts.lore:run
twistd = twisted.scripts.twistd:run
cftp = twisted.conch.scripts.cftp:run

So that it knows which module to load and which "main" function to execute.

I think you are right and that the Twisted setup.py should specify either entry_points or scripts but not both.

I think you are right that the meat should be in twisted.python.dist. It looks like I could make a sibling to getScripts() named getConsoleScripts or something.

I tried to write code to generate the entry points from the crud in trunk/bin/, but the names are not entirely consistent so that would require grepping the module name and function name out of the script in trunk/bin/. I suppose the best way to do it is to have an authoritative list of those names (pretty much like the one that my patch embeds into setup.py and then autogenerate the trunk/bin/ scripts from that?

Now for your very important objection: testing. This is a patch that doesn't seem testable by code that runs inside trial. It seems like to test it you need to have a buildbot step to install Twisted and then check whether trial and the others can be executed.

I am unsure how to go about writing such code for the use of the twisted project. It would be nice if the buildbot master.cfg were publicly readable -- perhaps in SVN -- so that everyone could see what I'm doing if I contribute a patch to the master.cfg. Exarkun and Radix (if I recall correctly) did something similar to test patches that I contributed to Nevow's build process. e.g.  http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall/builds/155/steps/shell_8/logs/stdio

Any other ideas about how to test this?

[glyph]

Replying to zooko:

I tried to write code to generate the entry points from the crud in trunk/bin/, but the names are not entirely consistent so that would require grepping the module name and function name out of the script in trunk/bin/. I suppose the best way to do it is to have an authoritative list of those names (pretty much like the one that my patch embeds into setup.py and then autogenerate the trunk/bin/ scripts from that?

Yes.

Now for your very important objection: testing. This is a patch that doesn't seem testable by code that runs inside trial. It seems like to test it you need to have a buildbot step to install Twisted and then check whether trial and the others can be executed.

It would be nice if distutils and/or setuptools supplied us with test harnesses that made it trivial to actually do an end-to-end test of everything, but I don't think that's necessary in this case.

We do have a builder that uses easy_install to install Twisted before running the tests:  http://buildbot.twistedmatrix.com/builders/debian-easy-py2.5-epoll

The pattern that appears to have been followed thus far seems to be " minimize untestable code", considering setup.py itself untestable. At least, I can't find any direct tests for setup.py. If you wanted to continue that, I'd be okay with it. The code which computed the list of entrypoints, which populated bin/ without setuptools, would be easily testable in isolation.

If you wanted to improve things beyond the status quo, I'd be happy, of course. The first thing that occurs to me is moving all of the code which actually computes the arguments to setup() into a module, so that we can inspect the complete list of arguments easily without invoking setup() itself; reducing setup.py to just a couple of lines to import two functions and invoke one with the result of another.

You could also write some test code for trial which did spawnProcess(.../setup.py install --prefix tempdir) once and kept the results around for a suite of tests to inspect the results.

I am unsure how to go about writing such code for the use of the twisted project. It would be nice if the buildbot master.cfg were publicly readable -- perhaps in SVN -- so that everyone could see what I'm doing if I contribute a patch to the master.cfg. Exarkun and Radix (if I recall correctly) did something similar to test patches that I contributed to Nevow's build process. e.g.  http://buildbot.divmod.org/builders/linux32-py2.5-nevowinstall/builds/155/steps/shell_8/logs/stdio

I don't know nearly enough about the buildbots to comment. It sounds like a good idea, though.