Opened 6 years ago

Closed 4 years ago

#7807 defect closed fixed (fixed)

extras_require options do not work with python3

Reported by: Chris Wolfe Owned by: Craig Rodrigues
Priority: normal Milestone:
Component: core Keywords:
Cc: kmike Branch: 7807-rodrigc-extrasrequire-py3
branch-diff, diff-cov, branch-cov, buildbot
Author: hawkowl, herrwolfe

Description

The extras_require options introduced with ticket #3696 fail when python3 is used. This can be reproduced by taking the following steps:

The following steps assuming python 3.3 is installed

# cd into root directory of twisted repostory
$ cd ./twisted

# create a python3 virtualenv
$ pyenv-3.3 venv

# activate the environment
$ . ./venv/bin/activate

# install an editable version of twisted from the local source with the `dev` option
$ pip install -e ".[dev]"

Obtaining file:///Users/chris/Code/twisted
    file twisted.py (for module twisted) not found
    file twisted/internet.py (for module twisted.internet) not found
    file twisted/internet/test.py (for module twisted.internet.test) not found
    file twisted/names.py (for module twisted.names) not found
    file twisted/names/test.py (for module twisted.names.test) not found
    file twisted/protocols.py (for module twisted.protocols) not found
    file twisted/protocols/test.py (for module twisted.protocols.test) not found
    file twisted/python.py (for module twisted.python) not found
    file twisted/python/test.py (for module twisted.python.test) not found
    file twisted/test.py (for module twisted.test) not found
    file twisted/trial.py (for module twisted.trial) not found
    file twisted/trial/test.py (for module twisted.trial.test) not found
    file twisted/web.py (for module twisted.web) not found
    file twisted/web/test.py (for module twisted.web.test) not found
  Exception:
  Traceback (most recent call last):
    File "/Users/chris/.pyenv/versions/3.3.5/lib/python3.3/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2501, in requires
      deps.extend(dm[safe_extra(ext)])
  KeyError: 'dev'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/chris/.pyenv/versions/3.3.5/lib/python3.3/site-packages/pip/basecommand.py", line 232, in main
      status = self.run(options, args)
    File "/Users/chris/.pyenv/versions/3.3.5/lib/python3.3/site-packages/pip/commands/install.py", line 339, in run
      requirement_set.prepare_files(finder)
    File "/Users/chris/.pyenv/versions/3.3.5/lib/python3.3/site-packages/pip/req/req_set.py", line 436, in prepare_files
      req_to_install.extras):
    File "/Users/chris/.pyenv/versions/3.3.5/lib/python3.3/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2504, in requires
      "%s has no such extra feature %r" % (self, ext)
  pip._vendor.pkg_resources.UnknownExtra: Unknown 15.0.0 has no such extra feature 'dev'

Attachments (1)

extras-require-py3-7807-1.patch (624 bytes) - added by Chris Wolfe 6 years ago.

Download all attachments as: .zip

Change History (13)

comment:1 Changed 6 years ago by Chris Wolfe

Several of the extra options are not yet python3 compatible.

  • The dev option fails because twistedchecker's setup.py file doesn't appear to be compatible with python3.
  • tls succeeds
  • conch succeeds
  • soap fails due to its dependency on wstools
  • serial succeeds
  • osx_platform, I couldn't try. The library states that is compatible with python3. But, as this install pyobjc and the soap options (in addition to tls, conch, and serial) I expect it to fail due to the wstools failure.
  • windows_platform - same issue as with osx_platform; the pypiwin32 library supports py3, but the option will fail due to wstools.

Changed 6 years ago by Chris Wolfe

comment:2 Changed 6 years ago by Chris Wolfe

Keywords: review added

The patch https://twistedmatrix.com/trac/attachment/ticket/7807/extras-require-py3-7807-1.patch adds the extras_require options to the setup3.py file. It makes no effort to remove python3 incompatible dependencies.

As of right now, not all of the options will successfully install on python3. As far as I can tell, the failing options are soap, dev, osx_platform and windows_platform.

The failures are

  • dev fails because of twistedchecker
  • soap fails because of wstools
  • osx_platform, windows_platform fail because of the soap option

If the patch is taken as is, users can install most of the dependencies. But, the install will fail if an option has dependencies that incompatible with python3.

Should a different set of options be available on py3? This seems workable, but it would require extra effort to maintain the list until all of the dependencies are py3 compatible.

Thank you.

comment:3 Changed 6 years ago by Adi Roiban

Keywords: review removed
Owner: set to Chris Wolfe

Many thanks for working on this and sorry for the delay.

args['extras_require']=_EXTRAS_REQUIRE

should be

args['extras_require'] = _EXTRAS_REQUIRE

The patch also needs a newsfile


I think that is important to have at least the .dev environment installable on py3 as otherwise, what is the point of having this patch and how can we test it in an automated waw ?

I will try to port twistedchecker on python3 as I think that we are close... I have already tried this but got stuck at FilePath, which now is ported.

For wstools there is this ticket https://github.com/pycontribs/wstools/issues/3

I think that we can skip wstools on python3, so yes, we should have different dependencies (subset) for Python3 as we also have support different features (a subset) for Python3.


In the same time I think that Twisted is too big... why not have the SOAP support as an external project ? But this is another discussion and another ticket.

comment:4 Changed 6 years ago by Adi Roiban

Keywords: review added

comment:5 Changed 6 years ago by Chris Wolfe

Keywords: review removed

Thanks for your review adiroiban.

I will work on your comments and add the review flag back once I have made your requested changes.

comment:6 Changed 5 years ago by kmike

Cc: kmike added

I think that is important to have at least the .dev environment installable on py3 as otherwise, what is the point of having this patch and how can we test it in an automated way ?

I was hit by this issue when trying to install twisted[tls] in Python 3.4, so IMHO it is useful even without [dev].

comment:7 Changed 5 years ago by hawkowl

Author: herrwolfehawkowl, herrwolfe
Branch: branches/extras-py3-7807

(In [46952]) Branching to extras-py3-7807.

comment:8 Changed 4 years ago by Craig Rodrigues

Branch: branches/extras-py3-78077807-rodrigc-extrasrequire-py3

comment:9 Changed 4 years ago by Craig Rodrigues

Keywords: review added

comment:10 Changed 4 years ago by Cory Benfield

Keywords: review removed
Owner: changed from Chris Wolfe to Craig Rodrigues

One really tiny nit of code review in the diff, but otherwise I'm happy with this. Resubmit for review when that's dealt with, and feel free to assign directly back to me.

comment:11 Changed 4 years ago by Cory Benfield

Given some further discussion on the ticket, I'm happy with this as-is. Feel free to merge.

comment:12 Changed 4 years ago by GitHub <noreply@…>

Resolution: fixed
Status: newclosed

In 40c4268:

Merge pull request #287 from twisted/7807-rodrigc-extrasrequire-py3

Author: rodrigc
Reviewer: Lukasa
Fixes: #7807

Fix on Python 3: pip install -e ".[dev]"

Note: See TracTickets for help on using tickets.