Opened 7 years ago

Last modified 2 years ago

#7477 enhancement assigned

replace all usage of pywin32 with cffi

Reported by: Glyph Owned by: Oliver Palmer
Priority: normal Milestone:
Component: core Keywords:
Cc: Adi Roiban, Frédéric Wang, Calum Lind Branch:
Author:

Description

Getting Python and the relevant development tools working on Windows is a big pain.

pywin32 really exacerbates this problem because it is resolutely stuck in a troubled period of Python's past:

  1. It's not present as a download in any form on PyPI, so pip install and easy_install don't work.
  2. They don't ship binary wheels, so installation takes a long time even if you've got a fully set up development environment.
  3. The EXE installers have some weird incompatibilities which means that you can't even cram them into a virtualenv with easy_install.
  4. Want to fix this to build your own wheels to work around this? If you don't have MSDN, too bad, you can't build it with the free tools and they aren't going to fix that. Maybe too bad even if you do, I haven't tried with a full install of VS2K8 Pro yet.

Given that we already depend on cffi, and all we're using pywin32 for is to call a couple of win32 APIs, it seems like we could drop the dependency entirely without an unreasonable amount of effort.

Change History (14)

comment:1 Changed 7 years ago by Glyph

For what it's worth, I have managed to build a binary wheel with just a couple of tweaks (changing os.environ["PATH"] in a pth file so that some DLLs that the PYDs depend on can be found). You can find it here.

comment:2 Changed 7 years ago by tritium21

To point 4: that isn't exactly accurate. You *CAN* build it with free tools, VS2K8Express is just not the tool to use IF, and ONLY IF you are trying to build 64bit binaries. You CAN use the compiler that comes with the windows 7 SDK. This ONLY effects mainline python builds of 2.6 - 3.2. 3.3 is built with 2K10, the express version of which comes with a 64 bit compiler. In order for them to fix the issue that 'they don't care about' is for Mark Hammond to buy Microsoft.

In other words, this is not a pywin32 issue, its a Microsoft issue, and only for the minority of users who use 64 bit versions of mainline Python.

EDIT: Ok, I have to redact part of this. I assumed, and everyone knows what that is an acronym for.

Anyways, mc.exe comes with the windows 7 sdk, which is the full compiler suit for windows, and it is free. so the main point still stands, that there are free tools that will compile this ...increasingly hunk of junk module... but the 2008 express editions are not one of those tools.

Last edited 7 years ago by tritium21 (previous) (diff)

comment:3 Changed 7 years ago by zooko

FWIW, replacing pywin32 with cffi is the approach we took in Tahoe-LAFS. We had quite a few issues back when Tahoe-LAFS used pywin32:

(Probably more tickets exist, but you get the idea.)

Then we finally removed our dependency on pywin32 altogether:

You might be interested to look at how we did it, by using ctypes to invoke the Windows native APIs and by avoiding Twisted's subprocess features in favor of the builtin "subprocess" module:

We were subsequently pretty disappointed when a new version of Twisted came out which added a hard dependency on pywin32, thus undoing some of the benefits we had gained. :-)

Our latest hackey idea is to switch from pywin32 to pypiwin32:

But if a new version of Twisted came out that did everything on Windows without using pywin32, that would be great for us!

comment:4 Changed 7 years ago by Glyph

pypiwin32 is definitely a temporary workaround while this can be done. We really don't touch that much of the API surface of pywin32.

comment:5 Changed 7 years ago by Glyph

Instead of just using the gross URL in the first comment on this ticket, you can, however, now `pip install pypiwin32` and the new extras support includes a reference to it.

comment:6 Changed 6 years ago by Oliver Palmer

Owner: set to Oliver Palmer
Status: newassigned

Going to go ahead and assign this to myself since I've started on this and have at least one module (python.lockfile) clear of pywin32. There's still a lot of work to be done with this so I'll be keeping this branch on github up to date as I work through the rest of the library:

https://github.com/twisted/twisted/compare/trunk...opalmer:windows-cffi

It's a bit of a mess right now but hopefully within the week or so I can get it cleaned up and working properly with the setup.py. Once that's done I might create an initial patch set for review. That might keep the reviews smaller and also make this a bit more incremental (unless someone has a better suggestion).

comment:7 Changed 6 years ago by Oliver Palmer

Broke off the first change into #7889. The branch in the link above was getting quite large so I've opted to start a little smaller first and replace a couple of functions that hopefully will be easier to test and review.

comment:8 Changed 6 years ago by Adi Roiban

Cc: Adi Roiban added

How is Twisted already dependent on cffi?

I see that #6831 is not merged yet.

Maybe we need a separate ticket to add cffi to dist.py and document that for Twisted it is planned to replace ctype usage with cffi

comment:9 in reply to:  8 Changed 6 years ago by Glyph

Replying to adiroiban:

How is Twisted already dependent on cffi?

It has no hard dependency on cffi yet. But you need cffi to use any of its TLS stuff.

comment:10 Changed 6 years ago by Glyph

Cc: Frédéric Wang added

From frederic.wang (spam-filtered, unfortunately):


Any update on this?

(I want to run buildbot slaves on MSYS2 shells and cffi seems much easier to build with mingw64 than pywin32)

comment:11 Changed 6 years ago by Frédéric Wang

Resaying what I wrote on #7889, although I'll probably be spam-filtered again. Enthought started to rewrite pywin32 with ctypes/cffi:

https://github.com/enthought/pywin32-ctypes https://pypi.python.org/pypi/pywin32-ctypes/0.0.1

"The default behaviour will try to use cffi if available and fall back to using ctypes."

The API seems very limited at the moment.

comment:12 Changed 3 years ago by Calum Lind

This is a gentle nudge to find out if this can be revived.

For the Deluge bittorrent client we use GTK and Twisted and packaging for Windows has got much harder recently with trying to move to GTK3 where the recommended install method is via MinGW. However Twisted is not available, primarily due to pywin32. We also use pywin32 but since Twisted depends on it, I've delayed migrating away from it.

I found that Oliver Palmer has created a new module pywincffi but it seems that development has stalled, although there is also a very rough PR based upon that work: https://github.com/twisted/twisted/pull/920

Any chance new progress could be made on this?

comment:13 in reply to:  12 Changed 3 years ago by Glyph

Replying to Calum Lind:

Hi Calum! Thanks for chiming in. Deluge is definitely an important Twisted user (hey, have you ever considered contributing to https://twistedmatrix.com/trac/wiki/SuccessStories?) and it would be great to get this fixed.

This is a gentle nudge to find out if this can be revived.

Absolutely! You may also be interested in https://twistedmatrix.com/trac/ticket/7945, which suggests putting all of this stuff in optional, rather than required, dependencies.

For the Deluge bittorrent client we use GTK and Twisted and packaging for Windows has got much harder recently with trying to move to GTK3 where the recommended install method is via MinGW. However Twisted is not available, primarily due to pywin32.

Is this because pywin32 can't be built against minGW's libraries or C runtime or something? More detail would be helpful. We stopped depending against my "pypiwin32" fork because pywin32 did start building binary wheels, which is supposed to solve this whole mess.

Looking at https://pypi.org/project/pywin32/#files right now I see wheels for 32bit / 64bit x 27 35 36 37, which seems like a reasonable collection. Sad face, no PyPy, but it doesn't sound like that's the problem you're having.

We also use pywin32 but since Twisted depends on it, I've delayed migrating away from it.

I found that Oliver Palmer has created a new module pywincffi but it seems that development has stalled, although there is also a very rough PR based upon that work: https://github.com/twisted/twisted/pull/920

Much as I like cffi, I would want to make very sure that we understand the exact build / integration problem that pywin32 is facing before attempting to pursue this particular strategy. Are these APIs even available under minGW? If they are, then why won't the existing binary wheels work? Is the issue merely that the pywin32 maintainers need to be asked to upload a wheel for a different platform tag?

Any chance new progress could be made on this?

Be the future that you want to see in the world :).

comment:14 Changed 2 years ago by Calum Lind

Cc: Calum Lind added

Be the future that you want to see in the world :).

I wish I has the requisite skills here (also I don't use Windows) to move this forward, other than pinging issues ;)

Is this because pywin32 can't be built against minGW's libraries or C runtime or something? More detail would be helpful.

Yes. Seems to be something to do with VC exception handling, although that was in 2015 so maybe the code has changed. This is last related issue: https://github.com/msys2/MINGW-packages/issues/751

it doesn't sound like that's the problem you're having.

Yes the problem relates to having everything available as MinGW packages so those MSVC compiled wheels are not usable with mingw-w64 Python.

Perhaps your knowledge betters understands the situation than I but certainly having twisted as a mingw packages would be super useful.

which suggests putting all of this stuff in optional, rather than required, dependencies.

What prevents it being merged? With no wheels on PyPi, need to use this website to install on Windows without compiling: https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted

Note: See TracTickets for help on using tickets.