[Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels

[Kyle Altendorf]

On 2018-09-07 01:58, Glyph wrote:
>> On Sep 5, 2018, at 5:44 AM, Kyle Altendorf <sda at fstab.net> wrote:
>> 
>> On 2018-09-05 03:25, Glyph wrote:
>>>> On Aug 29, 2018, at 10:13 AM, Kyle Altendorf <sda at fstab.net> wrote:
>>>> wheel links:
>>>> https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 
>>>> (and next comment)
>>>> Now that I've got the wheel builds happening I figured it'd be good 
>>>> to try them out on 'real' machines.  Turns out we get a failure on 
>>>> twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at 
>>>> least the two checks I've done so far (Windows and OSX).  I haven't 
>>>> done more than a cursory look at that yet, but it's on the list to 
>>>> understand and resolve.  More testing would of course be welcome.  
>>>> Real world, just trial Twisted's own tests, whatever would be 
>>>> appreciated if you are interested.
>>>> failures with wheels:
>>>> https://github.com/twisted/twisted/pull/1051#issuecomment-416977723
>>> It's not clear to me that we have any test environments that 
>>> accurately test what happens after you install from PyPI.  If we 
>>> could build these wheels and then actually run tests on installing 
>>> them that would be a big step forward in test fidelity!  But given 
>>> the current state of the art, I'd be happy if they just got built 
>>> from the same code that passed our tests in existing configurations.
>> 
>> cibuildwheel does have an option to provide a test command.  If we did 
>> that would we replace the existing tests or supplement?  If 
>> supplementing that will presumably be a significant increase in build 
>> times.  Especially if we did it for all wheels.
> 
> Definitely replace.  What would you want to even supplement them with?

I was referring to supplementing the existing src tests with the wheel 
tests.

> 
>> So, how much of a build time hit do we want to take?
> 
> I was thinking that for starters here the goal would just be build
> automation, independent of trying to get a more realistic test path;
> only doing these builds when pushing a relevant tag.  If it's easy to
> just switch over everything to this then... great, I guess? :)  But
> let's not gild the lily here.

Alrighty, another PR for swapping over tests from src to wheels.

I would tend to always build the wheels.  It's easier for people to test 
intermediate releases for bugs.  It makes a release 'less special' so 
there is less likely to be a latent issue with the process or artifacts. 
  Etc.  But, let's see what the build time hit is.

Travis: 4 minutes, about half the time of a single version test job
AppVeyor: 7 minutes, about the same as a single version test job
Circle OS X: 9 minutes, a bit over a single version test job
Circle Linux: 5 minutes, a bit under a single (OS X) version test job

Ok, so for about the cost of testing a single Python version we can get 
wheels.  Especially if we want to switch over to testing wheels instead 
of source, I'd plan on building wheels for every commit.

>> but sure, I didn't mean to argue against automation.
> 
> Could we do a test run by uploading wheels to, say,
> https://test.pypi.org <https://test.pypi.org/> first?

Shall the process be build wheel, upload to test PyPI, install from test 
PyPI, test...  :]  But sure, I could see ending up with a test PyPI 
upload at least for a tag/release.  Could technically do it for every 
build but that seems excessive, even given my comments above about 
exercising the process more often rather than less.

>> Here's one spot I did automatic deployment off of Travis to PyPI.
>> 
>> https://github.com/altendky/gitignoreio/blob/aeeb9546b6f27f411b191699d3625b4d0a08e7cf/.travis.yml#L11-L19
>> 
>> Pretty straightforward though I think it does the wheel build which 
>> isn't what we'd really want.
> 
> That's pretty cool.  I assumed you'd have to write a 'twine' command -
> Travis just has a 'deploy' provider called "pypi"?  What's actually
> happening behind the scenes there, I wonder!
> 
> (Also, you should probably upload an sdist as well as a bdist_wheel;
> there are cases where wheels aren't always what you want.)

Yeah, it's cool but then you have to know what is behind it...  for each 
CI host...  and how to make it use the files you want...  So I'd tend 
towards a twine call.  I made this silly little script the other day to 
handle this in another repo.

https://github.com/altendky/pyqt5-tools/blob/master/twineontag.py

>> I'd have to check on how to deploy existing wheels as well as those 
>> from OS X and Windows.  I'd suggest a separate PR?  I mean maybe it'd 
>> be easier to get one PR reviewed than two but that doesn't seem like a 
>> great reason to mash things together.
> 
> Smaller PRs are always easier to review - they just take more latency
> to build :).

Ok, I'll make separate tickets for:

* Testing wheels vs. src
* Automatic deployment to PyPI

Cheers,
-kyle