[Twisted-Python] Test coverage requirements

[Glyph Lefkowitz]

> On Feb 26, 2017, at 3:51 PM, Jean-Paul Calderone <exarkun at twistedmatrix.com> wrote:
> 
> Hi,
> 
> I'm looking at some <https://codecov.io/gh/twisted/twisted/pull/509> recent <https://codecov.io/gh/twisted/twisted/pull/432> trunk <https://codecov.io/gh/twisted/twisted/pull/652> commits <https://codecov.io/gh/twisted/twisted/pull/695> (also, others) that seem to have non-trivial untested code at at ReviewProcess <https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:Thingsyourbranchorpatchmustcontain>.  I can't tell if the codecov reports are wrong or if the development process documentation is wrong or if the commits just violate policy or (I guess) some mix of the three.
> 
> Can anyone shed any light on this?

Thanks, JP.  Your habit of post-hoc commit review really helps keep the process honest - I wish more folks (myself included) had more time for it!

I wasn't party to most of these changes, except for 432, where I can say based on local testing I was very clear that coverage was not in fact an issue, and codecov was misreporting it.  So I can't comment on most of them specifically.

Generally though, I can say that Codecov, while more reliable than coveralls, is unfortunately really flaky.  (Case in point, right now, clicking on all the URLs you posted here, I just see "AccessDeniedAccess Denied" followed by a long 64-bit identifier which I'm not pasting here for fear that it's some kind of github credential.)  Its general unreliability is the main reason we removed the hard coverage requirement that blocked merging.

That said, it has been improving and if it keeps improving at the rate it has been, I expect that we'd be able to put that coverage blocker back in in another 3-4 months.  Perhaps something to talk about at PyCon.

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20170227/ba74d304/attachment-0002.html>