Opened 12 years ago

Closed 2 years ago

#4374 enhancement closed wontfix (wontfix)

plugin for trial to user Ned Batchelder's coverage.py

Reported by: zooko Owned by: zooko
Priority: normal Milestone:
Component: trial Keywords:
Cc: Thijs Triemstra Branch:
Author:

Description

Brian Warner wrote this cool plugin in Tahoe-LAFS codebase and I extracted it out into a separate package so others could use it. It seems to be working fine:

http://tahoe-lafs.org/trac/trialcoverage/browser

Perhaps we could include it with Twisted or at least hyperlink to it from the trial docs.

Change History (11)

comment:1 Changed 12 years ago by zooko

http://tahoe-lafs.org/trac/tahoe-lafs/ticket/969 (contribute coverage plugin to Twisted)

comment:2 Changed 12 years ago by Jean-Paul Calderone

Owner: changed from Jonathan Lange to zooko

Can you describe why someone should care about this instead of being satisfied with the coverage support trial has already?

And should it really be a reporter plugin? Particularly if it's included in Twisted itself, and so there is no need to actually use the plugin mechanism. --reporter-bwverbose-coverage seems to make the particularly strong case against doing it this way.

comment:3 Changed 12 years ago by zooko

I've never used the coverage support that trial has already. I'm very happy with coverage.py. Apparently you, exarkun, have also used coverage.py recently. Could you then answer your own question?

I agree that the reporter plugin approach seems wrong, but I guess that's the only way I know how to do it without patching Twisted itself. Please let me know if there is a better way to plugin to trial.

comment:4 Changed 11 years ago by Jean-Paul Calderone

Can you describe why someone should care about this instead of being satisfied with the coverage support trial has already?

coverage.py supports branch coverage (with only a few minor issues). It also generates fancy html reports, supports merging coverage results from different runs, and is actively developed instead of a tur^Wmature, feature complete stdlib member.

Please let me know if there is a better way to plugin to trial.

Replacing (or augmenting) the implementation of the built-in --coverage option would be one approach. There's no support for coverage plugins because this doesn't seem like a common enough thing to want to write a plugin for, I suppose.

comment:5 Changed 9 years ago by Thijs Triemstra

Cc: Thijs Triemstra added

comment:6 Changed 9 years ago by Julian Berman

I definitely like coverage.py better than trial's coverage option, but perhaps a better question is, do we need this in trial when coverage.py already runs trial perfectly well?

Ned himself typically recommends using coverage to invoke the test runner. Can you describe some advantages to inverting the control here?

comment:7 Changed 9 years ago by Jean-Paul Calderone

Can you describe some advantages to inverting the control here?

It's not possible to meaningfully measure the coverage of trial itself using "coverage.py run trial ...".

However, "coverage.py run trial ..." works fine for everything else.

comment:8 Changed 9 years ago by Richard Wall

A few points:

  1. exarkun has recently told me that he thinks the current trial --coverage should be ditched (and possibly replaced) (ticket:6290#comment:5)
  1. Running python-coverage and getting the results always seems like a pain. Here's how I run it:
    coverage run --omit '/usr/*,_trial_temp/*' --branch ./bin/trial --reporter=bwverbose twisted.test.test_failure
    

Then to generate html report

coverage html -d coverage-html
firefox coverage-html/twisted_python_failure.html

So integrating that into trial --coverage could make it much more accessible.

  1. I've tried https://tahoe-lafs.org/trac/trialcoverage but it required some hacks to install and run it. I haven't been able to file a bug report because the tahoe trac won't let me complete my registration. Here are the patches in case others want to try it.

For compatibility with recent setuptools...

  • setup.py

    === modified file 'setup.py'
     
    1515except ImportError:
    1616    pass
    1717else:
    18  use_setuptools(download_delay=0)
    19  use_setuptools()

For compatibility with recent versions of coverage.py

  • trialcoverage/trialcoverage.py

    === modified file 'trialcoverage/trialcoverage.py'
     
    2626
    2727import setuptools
    2828
     29from coverage.config import CoverageConfig
     30
    2931class SummaryTextParseError(Exception): pass
    3032
    3133# These plugins are registered via twisted/plugins/trialcoveragereporterplugin.py .
     
    9698            linesplit = line.split()
    9799            missix = linesplit.index('Miss')
    98100            try:
    99                 brpartix = linesplit.index('BrPart')
     101                brpartix = linesplit.index('Branch')
    100102            except ValueError, le:
    101103                sys.stderr.write("ERROR, this tool requires a version of coverage.py new enough to report branch coverage, which was introduced in coverage.py v3.2.\n")
    102104                le.args = tuple(le.args + (linesplit,))
     
    112114
    113115    def __init__(self, coverage, show_missing=False, ignore_errors=False):
    114116        super(ProgressionReporter, self).__init__(coverage, ignore_errors)
    115         self.summary_reporter = CoverageSummaryReporter(coverage, show_missing=show_missing, ignore_errors=ignore_errors)
     117
     118        config=CoverageConfig()
     119        config.from_args(
     120            show_missing=show_missing,
     121            ignore_errors=ignore_errors)
     122        self.summary_reporter = CoverageSummaryReporter(
     123            coverage, config=config)
    116124
    117125    def coverage_progressed(self):
    118126        """ Returns 0 if coverage has regressed, 1 if there was no
     
    136144        if outfile is None:
    137145            outfile = SUMMARY_FNAME
    138146        outfileobj = open(outfile, "w")
    139         self.summary_reporter.report(morfs, omit=omit, outfile=outfileobj, include=include)
     147        self.summary_reporter.config.omit = omit
     148        self.summary_reporter.config.include = include
     149        self.summary_reporter.report(morfs, outfile=outfileobj)
    140150        outfileobj.close()
    141151
    142152        self.curunc, self.curpart = parse_out_unc_and_part(fileutil.read_file(SUMMARY_FNAME, mode='rU'))
  1. Having fixed those things, it does seem to produce quite nice summary results. Its especially nice to be able to see the coverage change between branches. (but maybe that belongs as part of a different tool eg twisted-checker)
[richard@zorin Twisted]$ trial --reporter=bwverbose-coverage twisted.test.test_usage
...
twisted.test.test_usage.ZshCompleterTestCase.test_usernames ... [OK]
Coverage results written to /home/richard/projects/Twisted/.coverage

-------------------------------------------------------------------------------
code coverage summary
There was no previous best code-coverage summary found.
Current coverage left 9277 total lines untested (6610 lines uncovered and 2667 lines partially covered).

-------------------------------------------------------------------------------
Ran 35 tests in 3.586s

PASSED (successes=35)

[richard@zorin Twisted]$ chbranch Twisted ugly-usage-wrapping-1601

[richard@zorin Twisted]$ unbranch Twisted
~/projects/Twisted/trunk$ svn status --quiet
C:
~/projects/Twisted/trunk$ svn up
C: Updating '.':
C: At revision 38202.
~/projects/Twisted/trunk$ svn merge --non-interactive /home/richard/projects/Twisted/branches/ugly-usage-wrapping-1601/@38131 /home/richard/projects/Twisted/branches/ugly-usage-wrapping-1601/@HEAD
C: --- Merging r38132 through r38202 into '.':
C: U    twisted/test/test_usage.py
C: U    twisted/python/usage.py
C: A    twisted/topfiles/1601.misc
[richard@zorin Twisted]$ trial --reporter=bwverbose-coverage twisted.test.test_usage
...
twisted.test.test_usage.ZshCompleterTestCase.test_usernames ... [OK]
Coverage results written to /home/richard/projects/Twisted/.coverage

-------------------------------------------------------------------------------
code coverage summary
code coverage improvement!
Previous best coverage left 9277 total lines untested (6610 lines uncovered and 2667 lines partially covered).
Current coverage left 9267 total lines untested (6600 lines uncovered and 2667 lines partially covered).

-------------------------------------------------------------------------------
Ran 36 tests in 3.617s

PASSED (successes=36)

richard@zorin Twisted]$ cat .coverage-results/summary.txt
...
trunk/twisted/trial/reporter                                                              503    472    100     98     5%
trunk/twisted/trial/runner                                                                373    301    126     99    20%
trunk/twisted/trial/util                                                                  170    136     54     40    21%
-------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                    8797   6600   2667   2267    23%

comment:9 Changed 9 years ago by Julian Berman

Running python-coverage and getting the results always seems like a pain. Here's how I run it:

That's more or less how I run it, though instead of --omit I just use --source=.. Coverage has a .coveragerc thing, so if you create one you can actually shorten that even more, and I have an alias for coverage run && coverage html.

comment:10 Changed 9 years ago by Jean-Paul Calderone

FWIW I think --source is probably to be preferred over --omit. In particular, note that this means you can very easily track and generate the coverage for just the single Twisted module you happen to be working with, eg --source twisted.web.http. This has a pretty big impact on the time it takes to generate the html as well.

Another potential consideration is that coverage.py run disttrial won't do anything very useful. This is a fringe use case right now, but ideally multi-process testing will be the default in the future, so eventually this will be important.

comment:11 Changed 2 years ago by Jean-Paul Calderone

Resolution: wontfix
Status: newclosed

It seems like the prevailing sentiment is that you should just use coverage run ... instead of driving coverage.py programmatically. Also, the trialcoverage plugin seems to have been abandoned upstream.

In the light of these facts, I'm going to close this as "wontfix". Of course this doesn't mean it would be bad to have such integration and if anyone is interested in putting in the work on this in the future they should feel free to re-start the discussion.

Note: See TracTickets for help on using tickets.