explain, document, and test trial's compatibility requirements for other test tools

[glyph]

The issue of trial needing to be compatible with other test runners is periodically raised, for example, in the comments on #1870. I believe Zope's test runner is the most popular of these, but I'm not sure.

It seems as though there is little information available for those of us who might want to make changes to trial as to what changes are acceptable and what changes are not, with respect to this requirement.

Are there tests within trial which verify this compatibility? It looks like test_pyunitcompat is for running pyunit tests with trial, not trial tests with pyunit (or pyunit-like frameworks). If they're somewhere else, where are they? If not, can some be written? With some dependency detection, perhaps, so we don't cause an earthquake on all the buildbots at once?

Is there documentation within trial which explains what the requirements of "other test runners" are?

What is the list of supported, external test runners? Is this documented anywhere?

If all of these things exist, then this ticket could be resolved simply by making them easier to find. Placing a link on any linked twistedmatrix.com page tends to push stuff to the top of google results pretty fast. In searching around for a variety of phrases,  this file in Zope3 SVN was the only thing I came across.

[exarkun]

Thanks for filing this ticket.

Since this is often given as a reason for some change to be made or not to be made to trial, resolving this is probably the highest priority trial ticket right now. Any other potential bug fix or enhancement runs the risk of encountering a constraint which isn't documented or generally known.

[jml]

Fair call.

Here is a rough draft:

• Respect the interfaces defined in stdlib unittest.py
• In particular, TestCase?.run(TestResult?()) should 'work' (i.e. complete synchronously, populate the result object etc)
• If trial foo.test_something.TestFoo.test_pass works, then TestFoo('test_pass').run(TestResult()) should also work, and vice versa.

[jml]

(In [21827]) Define ITestCase to be the interface that test frameworks expect of a test.

• Author: jml
• Reviewers: therve, exarkun
• Fixes #2898

ITestCase is re-purposed to define what you can expect from a test object. We use this interface to adapt pyunit tests to things usable in Trial.

In the process we also clarify what's needed for compatibility with doctest and pyunit.

This branch also fixes #1950.

Refs #2739.

[jml]

(In [21903]) Branching to 'trial-compatibility-doc-2739'

[exarkun]

The URLs could use U{} to make them links in the output. No need for re-review.

[jml]

(In [21934]) Document Trial's compatibility requirements.

• Author: jml
• Reviewer: exarkun
• Fixes #2739

Add information to the trial package docstring that starts to define the compatibility requirements for trial, giving examples.