Opened 12 years ago

Closed 12 years ago

#2739 enhancement closed fixed (fixed)

explain, document, and test trial's compatibility requirements for other test tools

Reported by: Glyph Owned by:
Priority: highest Milestone:
Component: trial Keywords:
Cc: Branch: branches/trial-compatibility-doc-2739
branch-diff, diff-cov, branch-cov, buildbot
Author: jml

Description

The issue of trial needing to be compatible with other test runners is periodically raised, for example, in the comments on #1870. I believe Zope's test runner is the most popular of these, but I'm not sure.

It seems as though there is little information available for those of us who might want to make changes to trial as to what changes are acceptable and what changes are not, with respect to this requirement.

Are there tests within trial which verify this compatibility? It looks like test_pyunitcompat is for running pyunit tests with trial, not trial tests with pyunit (or pyunit-like frameworks). If they're somewhere else, where are they? If not, can some be written? With some dependency detection, perhaps, so we don't cause an earthquake on all the buildbots at once?

Is there documentation within trial which explains what the requirements of "other test runners" are?

What is the list of supported, external test runners? Is this documented anywhere?

If all of these things exist, then this ticket could be resolved simply by making them easier to find. Placing a link on any linked twistedmatrix.com page tends to push stuff to the top of google results pretty fast. In searching around for a variety of phrases, this file in Zope3 SVN was the only thing I came across.

Change History (8)

comment:1 Changed 12 years ago by Jean-Paul Calderone

Priority: normalhighest

Thanks for filing this ticket.

Since this is often given as a reason for some change to be made or not to be made to trial, resolving this is probably the highest priority trial ticket right now. Any other potential bug fix or enhancement runs the risk of encountering a constraint which isn't documented or generally known.

comment:2 Changed 12 years ago by Jonathan Lange

Fair call.

Here is a rough draft:

  • Respect the interfaces defined in stdlib unittest.py
  • In particular, TestCase.run(TestResult()) should 'work' (i.e. complete synchronously, populate the result object etc)
  • If trial foo.test_something.TestFoo.test_pass works, then TestFoo('test_pass').run(TestResult()) should also work, and vice versa.

comment:3 Changed 12 years ago by Jonathan Lange

(In [21827]) Define ITestCase to be the interface that test frameworks expect of a test.

  • Author: jml
  • Reviewers: therve, exarkun
  • Fixes #2898

ITestCase is re-purposed to define what you can expect from a test object. We use this interface to adapt pyunit tests to things usable in Trial.

In the process we also clarify what's needed for compatibility with doctest and pyunit.

This branch also fixes #1950.

Refs #2739.

comment:4 Changed 12 years ago by Jonathan Lange

author: jml
Branch: branches/trial-compatibility-doc-2739

(In [21903]) Branching to 'trial-compatibility-doc-2739'

comment:5 Changed 12 years ago by Jonathan Lange

Keywords: review added
Owner: Jonathan Lange deleted

comment:6 Changed 12 years ago by Jean-Paul Calderone

Keywords: review removed
Owner: set to Jonathan Lange

The URLs could use U{} to make them links in the output. No need for re-review.

comment:7 Changed 12 years ago by Jonathan Lange

Resolution: fixed
Status: newclosed

(In [21934]) Document Trial's compatibility requirements.

  • Author: jml
  • Reviewer: exarkun
  • Fixes #2739

Add information to the trial package docstring that starts to define the compatibility requirements for trial, giving examples.

comment:8 Changed 9 years ago by <automation>

Owner: Jonathan Lange deleted
Note: See TracTickets for help on using tickets.