[Twisted-Python] GitHub Actions parallelism limit increase

[Kyle Altendorf]

On 2021-03-31 05:45, Adi Roiban wrote:

> On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, <sda at fstab.net> wrote:
> 
>> On 2021-03-30 15:24, Glyph wrote:
>>>> On Mar 30, 2021, at 7:57 AM, Kyle Altendorf <sda at fstab.net> wrote:
>>>> 
>>>> Hi All,
>>>> 
>>>> Has anyone contacted GitHub to see if they would be willing to
>>>> increase the parallelism limit in Actions?  My understanding is that
>>>> we maintain two CI systems (GitHub Actions and Azure Pipelines) for
>>>> the sake of more parallelism.  While perhaps worthwhile, this 
>>>> doesn't
>>>> seem fun.  Maybe GitHub would be willing to help out.
>>> 
>>> Not as far as I know. Do you want to give it a shot?
>> 
>> That was the plan.  Submitted.  I'll let you all know.
> 
> Thanks.
> 
> On a related note I am working to reduce the general duration of 
> Twisted CI jobs.
> 
> This should result in keeping the jobs slot busy for less time and 
> allowing more total jobs per day.
> 
> The first step was to enable parallel trial tests. Use both CPUs 
> available to the CI VMs.
> This was already done for Linux and MacOS.
> 
> Distributed trial is not yet supported on Windows, but I am working on 
> it.
> I already have a working distributed trial for Windows on my system.
> It needs various fixed and I have submitted smaller PR for review.
> 
> Just getting distrial to work on Windows is not all...as we need to 
> update and clean the  Twsited test to reduce the side effects.
> Linux and macOS are more relaxed in terms of file access and for 
> example you can delete a file, even if its opened by another process or 
> by the same process.
> Windows is not happy about that.
> 
> And then there are the 2 pypy jobs taking 30 minutes each - working to 
> fix it here https://github.com/twisted/twisted/pull/1543/
> 
> And then there are also general CI jobs improvements.
> 
> For example we can save about 40 seconds  if we decide to stop sending 
> coverage to coveralls and only use codecov.io
> Another example is save another 1 minute by stop using `tox --notests` 
> and replace it with something like 
> https://github.com/twisted/python-info-action to show the dependencies.
> With `tox --notest` we now create the wheels and install the 
> dependencies twice for each job.

Would `tox --skip-pkg-install` cut it for the second invocation?

https://github.com/ericaltendorf/plotman/pull/66/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR145

> There is an extra 1 minute required for each job to generate the 
> coverage XML file. We can download all the coverage artifacts and 
> generate the XML only once.

I did coverage collection from all jobs for pymodbus.  Looks like I 
still had the XML report generating in each job, but that could 
presumably be shifted.  Anyways, I'm sure several of us could do this 
off the top of our heads but in case it is of any interest here's what I 
did.  Or maybe you just point out something silly I did there and should 
learn from.  :]

https://github.com/riptideio/pymodbus/pull/592/files

Cheers,
-kyle