[Twisted-Python] retrial, the new-and-improved unittesting framework for twisted

[Jonathan Simms]

Hello twisted people,

Many of you know that I've been working on rewriting the
twisted-framework's unittesting program "trial".

This message is for those of you who don't ;)

I've nearly completely redesigned trial in terms of both it's
user-facing features, and it's internal workings. In initial testing,
it's shown a 4x speed increase over regular trial, running twisted's
tests in roughly 220s on a p3-1Ghz machine (as compared to 800+ seconds
on the same box).

here are some of the features you can look forward to (from the
twisted.trial.unittest docstring, so please excuse the epydoc markup):

  1. B{Trial now understands deferreds!}
    - There is no reason to use L{twisted.trial.util.wait} or
      L{twisted.trial.util.deferredResult}. Write your deferred handling
      code as you normally would, make your assertions in your callbacks
      and errbacks, then I{return the deferred from your test method}.
      Trial will spin the reactor (correctly) and will wait for the
      results before running the next test. This will allow developers to
      write more natural-looking tests for their asynchronous code.
    - there is a new attribute that has been introduced, C{.timeout}.
      Trial will wait a default 4 seconds for a result from a deferred
      that is returned from a test method. If you wish to make this value
      smaller or larger:

          >>> from twisted.trial.unittest import TestCase
          >>> from twisted.internet import defer
          >>> class MyTestCase(TestCase):
          ...     def testThatReturnsADeferred(self):
          ...         return defer.success('Hooray!')
          ...     testThatReturnsADeferred.timeout = 2.8
          ...
          >>>

      This would cause trial to wait up to 2.8 seconds (quite needlessly in
      this case) for the deferred to either callback or errback

  2. B{Trial is now 100% compatible with new-style classes and zope
     interfaces}
    - Some people (the maintainer included), have been bitten in the past
      by trial's mediocre support for new-style classes (classes which
      inherit from object). In v2.0, nearly all of the classes that
      comprise the framework inherit from object, so support is built-in.
      Whereas before the TestCase finding machinery used a test for
      inheritance from L{twisted.trial.unittest.TestCase}, the new
      mechanism tests that L{twisted.trial.interfaces.ITestCaseFactory}
      is supplied by your class B{type}. You can write a custom TestCase,
      and trial will detect it and use it as a class to test, if you do:

          >>> import zope.interface as zi
          >>> from twisted.trial.interfaces import ITestCaseFactory
          >>> from twisted.trial.interfaces import ITestCase
          >>> class MyTestCase(object):
          ...     zi.classProvides(ITestCaseFactory)
          ...     zi.implements(ITestCase)
          >>>

      Naturally, the class should actually provide an implementation of
      L{twisted.trial.interfaces.ITestCase}.
    - To avoid any possible conflicts (and to provide component
      de-registration), trial uses it's own private adapter registry, see
      L{twisted.trial.__init__} for details.
    - Trial makes use of zope.interface.Interfaces to allow flexibility
      and adaptation. All objects implement interfaces, and those
      interfaces are centralized and documented in
      L{twisted.trial.interfaces}.


  3. B{All assert* and fail* methods are now top-level functions of the
  unittest module}
    - Previously they were only available as instance-methods on the
      TestCase. You can now import all of the assert* and fail* variants
      and use them as functions. This will allow you to use this
      functionality in helper classes and functions that aren't part of
      your TestCase (plus less typing ;])
    - Note: these methods are no longer part of the ITestCase API, but
      are provided as a backwards-compatability to classes written to use
      the original TestCase class.


  4. B{The trial script now accepts a --reporter option}
    - This is to allow for custom reporter classes. If you want to run a
      trial process remotely, and gain access to the output, or if you
      would just like to have your reporting formatted differently, you
      can supply the fully-qualified class name (of a class that
      implements L{twisted.trial.interfaces.IReporter}) to --reporter,
      and trial will report results to your class.
    - The Reporter is now (almost) totally stateless. All stats on the
      test run are held in the TestSuite and are reported as necessary
      using the ITestStats interface. This allows for greatly simplified
      design in the Reporter implementation.
    - The Reporter API has been greatly simplified by changing the method
      signatures so that methods are called with a single object that can
      easily be adapted to provide all information necessary about a
      given test phase.


  5. B{Compatibility for PyUnit tests}
    - Trial now supports stdlib unittest.TestCase classes transparently.
      This functionality is unstable, and has not been heavily tested.
    - Note: Trial accomplishes this by monkey-patching unittest.TestCase
      in L{twisted.trial.__init__}.
    - Please report any bugs you find with this feature to the
      twisted-python mailing list


  6. B{Experimental support for doctests}
    - The trial script now supports a --doctest=[module] option. The
      argument is a fully-qualified module name, and trial will use a
      modified version of DocTestSuite to run the doctests it finds.
    - My support for doctests is broken when using Python 2.4-alpha3,
      hopefully, i'll get this fixed by the time the first major-release
      comes out.
    - Note: you cannot use .skip or .todo attributes with doctests, all
      tests will be reported as pass/fail
    - Please report any bugs you find with this feature to the
      twisted-python mailing list


  7. B{expectedAssertions} is no longer supported
    - it was just too difficult to make radix's clever
      deferred-doublecheck feature work with this code revision. With his
      permisison, this feature has been removed.


Trial's 'special' attributes:

  1. .todo attributes can either be set on the TestCase or on an
  individual test* method, and indicate that the test is expected to
  fail. New tests (for which the underlying functionality has not yet
  been added) should set this flag while the code is being written. Once
  the feature is added and the test starts to pass, the flag should be
  removed.

  2. Tests of highly-unstable in-development code should consider using
  .skip to turn off the tests until the code has reached a point where
  the success rate is expected to be monotonically increasing.

  3. Tests that return deferreds may alter the default timeout period of
  4.0 seconds by adding a method attribute C{.timeout} which is the
  number of seconds as a float that trial should wait for a result.


I am currently developing this in a branch in the twisted repository. I
would like to encourage anyone who is curious to check it out and play
with it and its new features. I'd like to make it as bug free as
possible before I merge it back into trunk, and the more eyes I can get
on it, the better.

you can check it out by doing
"svn co svn://svn.twistedmatrix.com/svn/Twisted/branches/slyphon/retrial-2"

and thank you for your support....

-Jonathan "slyphon" Simms