[Twisted-Python] GitHub Actions parallelism limit increase

[Adi Roiban]

On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, <sda at fstab.net> wrote:

>
>
> On 2021-03-30 15:24, Glyph wrote:
> >> On Mar 30, 2021, at 7:57 AM, Kyle Altendorf <sda at fstab.net> wrote:
> >>
> >> Hi All,
> >>
> >> Has anyone contacted GitHub to see if they would be willing to
> >> increase the parallelism limit in Actions?  My understanding is that
> >> we maintain two CI systems (GitHub Actions and Azure Pipelines) for
> >> the sake of more parallelism.  While perhaps worthwhile, this doesn't
> >> seem fun.  Maybe GitHub would be willing to help out.
> >
> > Not as far as I know. Do you want to give it a shot?
>
> That was the plan.  Submitted.  I'll let you all know.
>

Thanks.

On a related note I am working to reduce the general duration of Twisted CI
jobs.

This should result in keeping the jobs slot busy for less time and allowing
more total jobs per day.

The first step was to enable parallel trial tests. Use both CPUs available
to the CI VMs.
This was already done for Linux and MacOS.

Distributed trial is not yet supported on Windows, but I am working on it.
I already have a working distributed trial for Windows on my system.
It needs various fixed and I have submitted smaller PR for review.

Just getting distrial to work on Windows is not all...as we need to update
and clean the  Twsited test to reduce the side effects.
Linux and macOS are more relaxed in terms of file access and for example
you can delete a file, even if its opened by another process or by the same
process.
Windows is not happy about that.

And then there are the 2 pypy jobs taking 30 minutes each - working to fix
it here https://github.com/twisted/twisted/pull/1543/

And then there are also general CI jobs improvements.

For example we can save about 40 seconds  if we decide to stop sending
coverage to coveralls and only use codecov.io
Another example is save another 1 minute by stop using `tox --notests` and
replace it with something like https://github.com/twisted/python-info-action
to show the dependencies.
With `tox --notest` we now create the wheels and install the dependencies
twice for each job.

There is an extra 1 minute required for each job to generate the coverage
XML file. We can download all the coverage artifacts and generate the XML
only once.

-------------

Now the actual job duration will differ... I see the pip cache download
taking 2 seconds or 2 minutes.

But on Linux it takes about 4 minutes to run the tests (inside tox with
wheel creation)... and then from 2minutes and above is the extra overhead

Cheers

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20210331/a52ef5cc/attachment.htm>