[Twisted-Python] List of required builds before a merge

[Glyph]

> On Mar 12, 2018, at 4:59 AM, Adi Roiban <adi at roiban.ro> wrote:
> 
> Hi,
> 
> It is not clear to me what builders need to pass before we can merge something.
> 
> I expect that all supported "platforms" need to pass, but it is not
> clear what are the currently supported platforms.
> 
> We have this info in the wiki but it does not help.
> https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:Howtomergethechangetotrunk
> 
> In GitHub I can see Travis / Appveyor and OSX from Buildot as "Required"

These are marked as "required" because they're necessary, but not sufficient.

> Is that all?

Ideally all the supported buildbots should be passing.  It's a real shame that the offline buildbots do not report any status, because it makes it very easy to miss them.  (I personally did not realize the ramifications of the way buildbot repots status until I looked at https://buildbot.twistedmatrix.com/boxes-all?branch=trunk&num_builds=10 <https://buildbot.twistedmatrix.com/boxes-all?branch=trunk&num_builds=10> just now).

> If I check the "supported" group in Buildbot, I see many more builders.
> The problem is that a significant number of slaves are down and those
> builders are not available.

So, normally I'd say, like Jean-Paul did, that we should just get in touch with the maintainers of the buildbots in question.

But it seems the buildbots in question were the ones we had running on our donated Rackspace Cloud account.

Logging into the control panel for that account, literally all the servers except for the buildmaster (i.e. buildbot.twistedmatrix.com <http://buildbot.twistedmatrix.com/>) have been deleted.  Not just shut down, but, completely gone.  This is baffling to me.  I do not know who could have done this or why.  There does not appear to be an audit log I can consult.  Based on billing data, and consistent with the buildbot logs, it appears that this occurred some time in early January.

> Is Fedora still supported and required?

That's the hope.  Those buildbots appear to be online.

> I suggest to use GitHub "Required" marker to document what platforms are supported.

I want to agree with you.  However, our tests are not reliable or performant enough for this.

The "required" marker makes it impossible to merge changes without a passing status or an administrator override.  This has an unfortunate set of corollaries.  Assuming a non-administrator reviewer:

If a single builder has a temporary configuration issue and you're not an administrator, you can't merge any code.
Let's say the probably of an intermittent test failing is 50 to 1.  A 2% chance.  The probability of a test suite passing is 98%.  We have 36 supported builders.  The probability of all the builders passing for a successful run is then just 13%; roughly 1 in 10 valid branches will be able to land. (I think our probability is actually quite a bit better than this these days, but you get my drift.)
Even if a contributor can force all the builds to re-run (which requires special permissions, and thus needs to wait for a project member) getting a successful run on every builder could require 2 or 3 tries, which could be 2 or 3 hours of waiting just to get one successful run on a platform that you know is not relevant to the change you're testing.

Therefore keeping a small core set of "most pass" statuses and allowing for some human judgement about the rest is a practical necessity given the level of compute resources available to us.

> We don't have time to maintain the infrastructure, so I suggest to
> drop support for anything that is not supported by Travis and
> Appveyor.

My preference would be to simply drop all the buildbots which have been (for some reason) destroyed from the supported build matrix, since the buildbots are still covering a multiplicity of kernels and environments that travis and appveyor aren't.  But, I don't have the time to do much more than write this email, so if we have no other volunteers for maintenance, I will support your decision to tear down the buildbots for now.

Jean-Paul recently pointed out that CircleCI has much more performant macOS builds than Travis, so if someone were motivated to make that change but didn't want to keep maintaining hardware, that might be one way to go.

> I know that this might be disruptive.
> I think that we need it in order to raise awareness that supporting a
> platform is not easy.

I do hope that this will provoke some potential volunteers to come forward to help maintain our failing infrastructure.

> If someone (including me) cares about a platform they should find a way to help to project supporting that platform.

> What do you think?

I do hope that if you're going to make a change, you'll consider something slightly less drastic than blowing up the buildbots entirely :).  But with a dozen servers having just disappeared with no explanation, it's a course of action which at least makes sense.

-g

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20180312/4639b2ca/attachment-0002.html>