Opened 5 years ago

Last modified 4 weeks ago

#6032 defect new

make twisted.internet.endpoints importable on Windows when pywin32 is not installed

Reported by: tray Owned by:
Priority: high Milestone:
Component: core Keywords:
Cc: davidsarah Branch:
Author:

Description (last modified by Jean-Paul Calderone)

Using Twisted 12.2, twisted.internet.endpoints cannot be imported on Windows unless pywin32 is available:

 Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "twisted\web\client.py", line 21, in <module>
    from twisted.internet.endpoints import TCP4ClientEndpoint, SSL4ClientEndpoint
  File "twisted\internet\endpoints.py", line 28, in <module>
    from twisted.internet import stdio
  File "twisted\internet\stdio.py", line 28, in <module>
    from twisted.internet import _win32stdio
  File "twisted\internet\_win32stdio.py", line 7, in <module>
    import win32api
ImportError: No module named win32api

As demonstrated by this traceback, this also affects any module that depends on endpoints. The pywin32 dependency is only necessary for stdio; other endpoints should be usable even if it is not installed.

This likely requires a Windows version of the no-modules builder to be set up first - see <https://bugs.launchpad.net/twisted-buildbot-configuration/+bug/1059240>.

Change History (15)

comment:1 Changed 5 years ago by Jean-Paul Calderone

Description: modified (diff)
Priority: highnormal
Summary: Either making pywin32 a requirement, or don't have endpoints break when it is missingmake twisted.internet.endpoints importable on Windows when pywin32 is not installed
Type: regressionenhancement

comment:2 Changed 5 years ago by Jean-Paul Calderone

Description: modified (diff)

comment:3 Changed 4 years ago by davidsarah

Cc: davidsarah added

This would fix Tahoe-LAFS ticket https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2028 .

comment:4 Changed 3 years ago by zooko

For the upcoming Tahoe-LAFS v1.11 release, we're working-around this issue by requiring Twisted < 12.2.0:

https://github.com/tahoe-lafs/tahoe-lafs/commit/d888b28b715020e879a52f5159b6e14c7dcab6b8

comment:5 Changed 3 years ago by zooko

This interacts with https://github.com/twisted/nevow/issues/43 (why does Nevow depend on Twisted >= 13.0?).

comment:6 Changed 3 years ago by zooko

I suppose switching from pywin32 to new fork "pypiwin32" https://pypi.python.org/pypi/pypiwin32 (which exists because of this reason — http://sourceforge.net/p/pywin32/feature-requests/110/ , http://sourceforge.net/p/pywin32/bugs/669/ ) would solve this issue.

comment:7 Changed 16 months ago by Glyph

Priority: normalhigh
Type: enhancementdefect

OK, I guess the Tahoe problem here is recently resolved, so the original reporters are probably not motivated to fix it; but, wow, this must be a terrible onboarding experience for windows users. We should definitely fix it.

comment:8 Changed 16 months ago by Glyph

Now that we test on appveyor, this (specifically, the no-modules builder) could be submitted as a regular Twisted ticket, without needing a buildbot configuration change.

comment:9 Changed 7 weeks ago by oTree-org

Would be really great to have this fixed. I maintain a web app framework that is based on Django Channels. I can't upgrade to the latest Channels because it depends on twisted.internet.endpoints, which relies on pypiwin32. I have ruled out adding a dependency on pypiwin32 because it creates too many installation/deployment complications for my users.

Here is an issue someone logged in Django Channels that also references this issue: https://github.com/django/channels/issues/498

comment:10 Changed 7 weeks ago by Glyph

Thanks for the update. If you would like to contribute a fix, please have a look at the development process documentation.

Please just contribute a change to appveyor.yml that runs a tox environment without "alldeps" in its name so that we have a build tracking what happens without pywin32; at that point it should be relatively easy to add the try/except around the relevant import.

comment:11 Changed 7 weeks ago by Glyph

We would absolutely love to support Django Channels' use of Twisted, and I'm sorry I don't have the time personally to do this fix.

comment:12 Changed 7 weeks ago by oTree-org

I wrapped the import in a simple try/except, and all Django Channels tests are passing.

In appveyor.yml, line 44, add this:

- PYTHON_HOME: C:\\PYTHON36-x64
  PYTHON_VERSION: "3.6"
  PYTHON_ARCH: "64"
  TOXENV: py36-nodeps-withcov-windows,codecov-publish
  TWISTED_REACTOR: "select"

In src/twisted/internet/endpoints.py, line 41, change this:

from twisted.internet.stdio import StandardIO, PipeAddress

To this:

try:
    from twisted.internet.stdio import StandardIO, PipeAddress
except ImportError:
    # fallback if pypiwin32 is not installed
    StandardIO = None
    PipeAddress = None

Also, this command passes (but it passed even before I made the change):

trial twisted.test.test_internet

Not sure how to really test or how to use AppVeyor. I'm totally new to Twisted. This is all the time I was able to spend on this for now. Leaving this here in case someone with more Twisted knowledge has a chance.

comment:13 in reply to:  9 Changed 7 weeks ago by mark williams

Replying to oTree-org:

Would be really great to have this fixed. I maintain a web app framework that is based on Django Channels. I can't upgrade to the latest Channels because it depends on twisted.internet.endpoints, which relies on pypiwin32. I have ruled out adding a dependency on pypiwin32 because it creates too many installation/deployment complications for my users.

Hi oTree-org! First, thanks for improving Twisted :)

I'm curious to know why pypiwin32 complicates installation and deployment. It's available as a wheel for Python 3.6. Would wheels for other Python versions make it less of an issue? Or is it just that it's a conditional transitive dependency - your framework depends on Django channels depends on Twisted, and now Windows users have remember to install pypiwin32 on Twisted's behalf?

I ask because we're considering moving to pywincffi for many reasons including ease of packaging, and if "making Twisted easier for Django channels" is one of those reasons, then I can prioritize that switch.

On the other hand, if any additional transitive dependency is a problem, then we need graceful degradation along the lines of what you've written in a later comment. We might need that anyway, because, as Glyph said, this is a really bad experience for Windows users.

comment:14 Changed 6 weeks ago by oTree-org

Thanks for the reply! I'm fine with just supporting Python 3.6. Rather, the issue is about adding a complex dependency that is platform specific.

pywincffi definitely looks more manageable than pypiwin32, which is huge and doesn't install on non-Windows systems (which breaks the ability to "pip freeze" and push to Heroku, collaborate with others, etc; many of my framework's users are novice programmers so they have difficulty troubleshooting these things).

But it would be good to eliminate unnecessary dependencies. Django channels already installs a sizable set of dependencies:

asgiref
attrs
autobahn
Automat
channels
constantly
daphne
hyperlink
incremental
six
Twisted
txaio
zope.interface

With the standard Redis backend, add:

asgi-redis
msgpack-python
redis

I would like to avoid making the list longer, in order to keep my project lightweight, and to be able to understand what each part is doing. In particular, pywincffi+cffi are pretty complex, low-level, and platform-specific (which means I need to do more testing, and anticipate what happens if a Mac/Linux user accidentally installs it, etc).

At the end of the day, pywincffi would probably be OK, but the import cleanly avoids the dependency altogether, which is why it is more appealing to me :)

Last edited 6 weeks ago by oTree-org (previous) (diff)

comment:15 in reply to:  14 Changed 4 weeks ago by mark williams

Replying to oTree-org:

Thanks for the reply! I'm fine with just supporting Python 3.6... ...In particular, pywincffi+cffi are pretty complex, low-level, and platform-specific (which means I need to do more testing, and anticipate what happens if a Mac/Linux user accidentally installs it, etc).

PEP 508 markers allow you to restrict dependencies to specific platforms:

(linux)$ pip install 'pywincffi ; platform_system == "Windows"'
Ignoring pywincffi: markers 'platform_system == "Windows"' don't match your environment

I did a little research with the PyPI BigQuery dataset to determine if people are using version of pip and setuptools that support PEP 508. Since you're OK supporting only 3.6, I limited the query to that version of Python:

SELECT
  details.installer.name,
  details.installer.version,
  COUNT(*) as total_downloads
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    TIMESTAMP("20160114"),
    CURRENT_TIMESTAMP()
  )
WHERE
  REGEXP_MATCH(details.python, r"^3.6")
GROUP BY
  details.installer.name,
  details.installer.version
ORDER BY
  total_downloads DESC

Here were the results as of today:

Rowdetails_installer_namedetails_installer_versiontotal_downloads
1pip9.0.1262450316
2setuptools28.8.03499929
3setuptools36.0.11606610
4pip8.1.21470281
5setuptools27.2.0848139
6setuptools35.0.2825094
7pip8.1.1757826
8setuptools36.2.0734394
9setuptools36.5.0717401

All of these versions of pip and setuptools support PEP 508 markers. I think you'd be safe adding them to your project's dependencies so that pypiwin32 only gets installed on Windows.

At the end of the day, pywincffi would probably be OK, but the import cleanly avoids the dependency altogether, which is why it is more appealing to me :)

I'll look into pywincffi. But in the meantime, let us know if PEP 508 markers help the situation!

Note: See TracTickets for help on using tickets.