Opened 6 years ago

Closed 5 years ago

#6539 enhancement closed fixed (fixed)

Support building, packaging, and installing the Python 3 compatible parts of Twisted using setup.py instead of setup3.py

Reported by: Hynek Schlawack Owned by:
Priority: normal Milestone: Python-3.x
Component: core Keywords:
Cc: Hynek Schlawack Branch: branches/python3-setup-6539-2
branch-diff, diff-cov, branch-cov, buildbot
Author: itamarst

Description (last modified by Jean-Paul Calderone)

setup3.py was always intended as a temporary measure. Its functionality should be part of setup.py, as much as that makes sense. As a start, it should be possible to build and install the Python 3 compatible parts of Twisted using setup.py run using Python 3. This means that this sequence of commands:

$ python3.3 setup.py sdist
$ cd dist
$ tar xf Twisted-A.B.C.tar.gz
$ cd Twisted-A.B.C
$ python3.3 setup.py install

should produce an installation which passes as many tests when run on Python 3.3 as pass from a source checkout.

Change History (28)

comment:1 Changed 6 years ago by therve

Are you sure setup3.py is the problem? I thought #6228 was the problem.

comment:2 Changed 6 years ago by Hynek Schlawack

Cc: Hynek Schlawack added

Ok it seems it was a red herring and a pure OS X problem. pip install on Linux seems to work fine. I’ll let this open since the separation doesn’t make sense but I’m opening another ticket for the original problem.

comment:3 Changed 6 years ago by therve

You're right, pip doesn't know about setup3.py apparently. Merging would fix the requirement problem also maybe.

comment:4 Changed 6 years ago by Jean-Paul Calderone

Description: modified (diff)
Summary: setup3.py should be merged into setup.pySupport building, packaging, and installing the Python 3 compatible parts of Twisted using setup.py instead of setup3.py

comment:5 Changed 6 years ago by Jean-Paul Calderone

$ tar xf Twisted-A.B.C.tar.gz

It might be worth thinking a little bit about this step.

Will `python2.7 setup.py sdist; python3.3 setup.py sdist´ generate two artifacts with the same name but different contents? What will happen if someone installs the python3.3 version on python2.7?

comment:6 Changed 6 years ago by Hynek Schlawack

Yes, upon cursory inspection I see that we produce different sdists for Python 2.7 and 3.3. In my understanding that’s very wrong™.

It’s a lot more work but we should really fix this, it’s technical debt that will hurt us in any case eventually.

comment:7 Changed 6 years ago by Itamar Turner-Trauring

At the very least I think it's worth being able to *install* with setup.py on Python 3. Right now it's not possible to pip install on Python 3.3. Beyond simply installing Twisted, this means doing things like automated Travis-CI testing of Twisted projects that work on both Python 2 and 3 is difficult (my personal use case - Crochet).

I propose limiting this ticket to support building and installing from setup.py on Python 3, and for now have sdist give an error when run on Python 3.

comment:8 Changed 6 years ago by Jean-Paul Calderone

I propose limiting this ticket to support building and installing from setup.py on Python 3, and for now have sdist give an error when run on Python 3.

This seems reasonable. The challenge is that doing this kind of distutils customization always ends up about five times harder than you expect (even taking this into consideration) and breaks at least one tool that someone was relying on.

comment:9 Changed 6 years ago by itamarst

Author: itamarst
Branch: branches/python3-setup-6539

(In [40569]) Branching to 'python3-setup-6539'

comment:10 Changed 6 years ago by Itamar Turner-Trauring

Well, it "works" if manually tried. Not sure how to unit test it though... or maybe we don't care to? How much coverage of installation do the buildslaves give us?

I'm going to point my Crochet Travis-CI config at this branch sometime soon, and see if that works.

comment:11 Changed 6 years ago by Thijs Triemstra

Milestone: Python-3.x

Adding the Python 3.0 milestone.

comment:12 Changed 6 years ago by Thijs Triemstra

Tested the branch a little using virtualenv with python3.3:

thijs@aspiration:~/Twisted/branches/python3-setup-6539$ mkvirtualenv twisted-6539 --python=python3.3
Running virtualenv with interpreter /usr/local/bin/python3.3
Using base prefix '/usr/local'
New python executable in twisted-6539/bin/python3.3
Also creating executable in twisted-6539/bin/python
Installing distribute......................................................................................................................................................................................................................................................................................................................................................................................................done.
Installing pip................done.
thijs@aspiration:~/Twisted/branches/python3-setup-6539$ workon twisted-6539
(twisted-6539)thijs@aspiration:~/Twisted/branches/python3-setup-6539$ python --version
Python 3.3.2

Instead of python3.3 setup.py install I tried python3.3 setup.py develop:

(twisted-6539)thijs@aspiration:~/Twisted/branches/python3-setup-6539$ python3.3 setup.py develop
running develop
running egg_info
creating Twisted.egg-info
writing dependency_links to Twisted.egg-info/dependency_links.txt
writing requirements to Twisted.egg-info/requires.txt
writing top-level names to Twisted.egg-info/top_level.txt
writing Twisted.egg-info/PKG-INFO
writing manifest file 'Twisted.egg-info/SOURCES.txt'
file twisted.py (for module twisted) not found
file twisted/internet.py (for module twisted.internet) not found
file twisted/internet/test.py (for module twisted.internet.test) not found
file twisted/internet/util.py (for module twisted.internet.util) not found
file twisted/names.py (for module twisted.names) not found
file twisted/names/test.py (for module twisted.names.test) not found
file twisted/protocols.py (for module twisted.protocols) not found
file twisted/protocols/test.py (for module twisted.protocols.test) not found
file twisted/python.py (for module twisted.python) not found
file twisted/python/test.py (for module twisted.python.test) not found
file twisted/test.py (for module twisted.test) not found
file twisted/trial.py (for module twisted.trial) not found
file twisted/trial/test.py (for module twisted.trial.test) not found
file twisted/web.py (for module twisted.web) not found
file twisted/web/test.py (for module twisted.web.test) not found
writing manifest file 'Twisted.egg-info/SOURCES.txt'
running build_ext
Creating /home/thijs/.virtualenvs/twisted-6539/lib/python3.3/site-packages/Twisted.egg-link (link to .)
Adding Twisted 13.1.0-r40575 to easy-install.pth file

Installed /home/thijs/Twisted/branches/python3-setup-6539
Processing dependencies for Twisted==13.1.0-r40575
Searching for zope.interface>=4.0.2
Reading http://pypi.python.org/simple/zope.interface/
Best match: zope.interface 4.0.5
Downloading https://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.5.zip#md5=caf26025ae1b02da124a58340e423dfe
Processing zope.interface-4.0.5.zip
Writing /tmp/easy_install-vb98tn/zope.interface-4.0.5/setup.cfg
Running zope.interface-4.0.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-vb98tn/zope.interface-4.0.5/egg-dist-tmp-11vy74
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
Adding zope.interface 4.0.5 to easy-install.pth file

Installed /home/thijs/.virtualenvs/twisted-6539/lib/python3.3/site-packages/zope.interface-4.0.5-py3.3-linux-x86_64.egg
Searching for distribute==0.6.34
Best match: distribute 0.6.34
Processing distribute-0.6.34-py3.3.egg
distribute 0.6.34 is already the active version in easy-install.pth
Installing easy_install script to /home/thijs/.virtualenvs/twisted-6539/bin
Installing easy_install-3.3 script to /home/thijs/.virtualenvs/twisted-6539/bin

Using /home/thijs/.virtualenvs/twisted-6539/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg
Finished processing dependencies for Twisted==13.1.0-r40575

This installed the dependencies and twisted but there are some errors as you can see (that seem to be harmless?):

file twisted.py (for module twisted) not found
file twisted/internet.py (for module twisted.internet) not found
file twisted/internet/test.py (for module twisted.internet.test) not found
file twisted/internet/util.py (for module twisted.internet.util) not found
etc

The same errors show up when it's installed using pip -e:

(twisted-6539)thijs@aspiration:/tmp/twisted$ pip install -e ~/Twisted/branches/python3-setup-6539
Obtaining file:///home/thijs/Twisted/branches/python3-setup-6539
  Running setup.py egg_info for package from file:///home/thijs/Twisted/branches/python3-setup-6539
    
    file twisted.py (for module twisted) not found
    file twisted/internet.py (for module twisted.internet) not found
    file twisted/internet/test.py (for module twisted.internet.test) not found
    file twisted/internet/util.py (for module twisted.internet.util) not found
    file twisted/names.py (for module twisted.names) not found
    file twisted/names/test.py (for module twisted.names.test) not found
    file twisted/protocols.py (for module twisted.protocols) not found
    file twisted/protocols/test.py (for module twisted.protocols.test) not found
    file twisted/python.py (for module twisted.python) not found
    file twisted/python/test.py (for module twisted.python.test) not found
    file twisted/test.py (for module twisted.test) not found
    file twisted/trial.py (for module twisted.trial) not found
    file twisted/trial/test.py (for module twisted.trial.test) not found
    file twisted/web.py (for module twisted.web) not found
    file twisted/web/test.py (for module twisted.web.test) not found

[...]

    Installed /home/thijs/Twisted/branches/python3-setup-6539
Successfully installed zope.interface Twisted
Cleaning up...

Not sure why though...

comment:13 Changed 6 years ago by Itamar Turner-Trauring

Keywords: review added

Not sure what else to do, so putting up for review in the hopes someone has better sense if we have buildbot coverage for running setup.py on Python 2.

comment:14 Changed 6 years ago by Jean-Paul Calderone

Keywords: review removed
Owner: set to Itamar Turner-Trauring

Is http://buildbot.twistedmatrix.com/builders/debian-easy-py2.6-epoll/builds/1931/steps/shell_2/logs/stdio sort of like the kind of test coverage you're thinking of?

It's not "setup.py install" but it's as close as we have (perhaps we *should* have something like this builder but using "setup.py install" instead; and if people care about "pip" then I guess a builder doing "pip whatever").

comment:15 Changed 6 years ago by Hynek Schlawack

JFTR, that "whatever" would be "install path-of-result-of-sdist". That would be a great thing to have as part of our CI to ensure our sdists are always installable.

comment:16 Changed 6 years ago by Itamar Turner-Trauring

What's the implied next step in the review?

comment:17 Changed 6 years ago by Hynek Schlawack

I wasn’t the reviewer but if I were, I would ask for the buildbot integration I mentioned otherwise there’s no way our packaging isn’t broken. Install the sdist and at least try to import twisted inside.

Since I hope we’ll be able to do more work on Twisted’s packaging in the near future, this would be most useful. Can we pester Tom about this?

Sorry for heckling.

comment:18 Changed 6 years ago by Itamar Turner-Trauring

We don't use sdist in our packaging. Thus the scope of this ticket, as far as I am personally concerned, is being able to setup.py install with Python 3, and ideally pip/easy_install under Python 3 the tarball of a release that has been uploaded PyPI. Otherwise it's never going to happen.

We can certainly have separate tickets for other useful steps to take.

comment:19 Changed 6 years ago by Hynek Schlawack

I’m sorry, I didn’t know that we do our own thing. So for the sake of this ticket, I agree with you.

comment:20 Changed 5 years ago by Hynek Schlawack

I’ve done some testing:

  • your approach makes python setup.py install work on Python 3.3.
  • in order to make pip/easy_install work, we have to ship setup3.py since setup.py refers to it and non-wheel installs encompass arbitrary code execution.

I’ve successfully tried adding a MANIFEST.in with the sole line setup3.py. The .tar.gz that gets created now using sdist is indeed pip-installable under Python 3.3. I’d therefore suggest that route.

comment:21 Changed 5 years ago by Itamar Turner-Trauring

Keywords: review added
Owner: Itamar Turner-Trauring deleted

I added a test to verify setup3.py will be included in packages, and was verified that I was able to pip install a checkout.

comment:22 Changed 5 years ago by Hynek Schlawack

This may sound really stupid, but is there a way to test-build a Twisted tar ball without involving svn branches (i.e. http://twistedmatrix.com/trac/wiki/ReleaseProcess#Howtodoapre-release) ? Because setup.py sdist is still missing setup3.py and I’m not sure how I can hand-verify that it will do what we want.

Tests are great and pip install of a checkout will obviously work because the setup3.py is right there, but I’d really like a simple way to verify it in “real life”. Am I missing something?

comment:23 Changed 5 years ago by Itamar Turner-Trauring

bin/admin/build-tarballs I would guess.

comment:24 Changed 5 years ago by Hynek Schlawack

Keywords: review removed

Ouch, I read the error message wrong, all I needed was a svn checkout.

  1. I have forced a build, the bots are green/fail unrelatedly.
  2. I have manually verified:
    1. setup3.py is package inside of Twisted-A.B.C.tar.bz2
    2. Twisted-A.B.C.tar.bz2 is pip-installable.
    3. Twisted-A.B.C.tar.bz2 is not inside of TwistedCore-A.B.C.tar.bz2 whose gravity I’m not sure of – I can live with only the whole package being pip-installable under Python 3. This problem will resolve itself eventually.
    4. Since 'admin/run-python3-tests' has been packaged inside of setup3 previously, it’s in neither. I would suggest to add it. Ideally add a test too.
  3. Code:
    1. Adding args['classifiers'] = ["Programming Language :: Python :: 3.3"] (and "Programming Language :: Python :: 3") belongs into the main setup.py since the whole point of this ticket is to make our sdists Python 3-capable. :)
    2. Mention pip in the news fragment?
  4. Next steps:
    1. I would appreciate a ticket to modify our buildbots to make sure our Tarballs are pip-installable as exarkun suggested. Maybe Tom can comment on how much work that would be.
    2. I trust you can decide on the gravity of the things I’ve pointed out yourself so I leave it to you to either fix and merge or fix and put it out to review again. I will try to review swiftly if you decide for the latter.

Thanks for working on this, I can’t wait to test structlog’s Twisted module in CI. :)

comment:25 Changed 5 years ago by Hynek Schlawack

Ugh, 2.3 should have been

“setup3.py” is not inside of TwistedCore-A.B.C.tar.bz2

of course.

comment:26 Changed 5 years ago by Itamar Turner-Trauring

comment:27 Changed 5 years ago by itamarst

Branch: branches/python3-setup-6539branches/python3-setup-6539-2

(In [41377]) Branching to 'python3-setup-6539-2'

comment:28 Changed 5 years ago by itamarst

Resolution: fixed
Status: newclosed

(In [41379]) Merge python3-setup-6539-2. Make setup.py install on Python 3.

Author: itamar Reviewer: exarkun, hynek Fixes: #6539

setup.py install and pip install now work on Python 3.3.

Note: See TracTickets for help on using tickets.