[Twisted-Python] Thoughts about testing

[Glyph Lefkowitz]

On Wed, 2005-10-26 at 04:56 -0700, David Ripton wrote:

> Things I particularly like about py.test, vs. unittest:

Can we compare to trial as it stands please?  I don't think anyone is
seriously suggesting stdlib unittest for Twisted, and I am pretty sure
trial has already subsumed every interesting feature from unittest
itself :)

> 1. assert, not self.failUnlessAllThisUglyCamelCaseActuallyKillsMe()

This is cute - more in the way it deals with printing appropriate
context than the fact that you actually use 'assert' - but it's a shame
that they resorted to a language hack rather than a simple function like
'test()' or something.  Using 'assert' makes it impossible to test with
-O or -OO.  I personally don't use those options, but some people do,
and they should be testable.

> 2. You can use module-level test functions.  No need to make a test
> class unless it's actually helpful.  (Yay Java.)

Handy in some cases, but from what I've seen in my brief survey of
py.test-based tests the only thing this does is make it more of a
convention to smear unit test state across a bunch of shared globals ...

> 3. Tests execute in predictable order.

... and then since there is now global state smeared across your whole
module, the tests have to run in an exact order because each one sets up
things for the next, rather than proper setUp/tearDown encapsulation.

> 4. setup_module / setup_class / setup_method.  Handy for reusing
> database connections.

Technically trial has similar features, but they were broken so horribly
for so long that it became the common wisdom not to use them.  I have to
assume this has recently changed?

> 5. A separate test runner binary, so you don't need __main__ boilerplate
> in every test module, and you can put tests in the same module as a 
> script (if you don't care about them being found automatically).

Trial has this already too.

> 6. You can fill your tests with print statements, and only see their
> output if the test fails (or you pass the -s option).

This is like a replication of trial's AND unittest's worst feature - the
fact that tracebacks are deferred until execution is finished, so tests
which hang intermittently are un-debuggable.  I believe it is only for
the buildbot that -e is not the default.  Speaking of which, let me go
update TwistedEmacs...