trial reports errors when it shouldn't

[Drew Smathers]

I noticed some odd behaviors with trial where test reports result in errors even though the underlying errors are handled. The behaviors I've noticed are as follows:

1. If I log an error somewhere in code with log.err, then the test results in an error
2. Errors inside a DeferredList result in a test error (even if the failure is handled)

Here is an example which demonstrates the above behaviors:

from twisted.internet import defer
from twisted.python import log
from twisted.trial.unittest import TestCase


class SomeComponent:
    fails = 0

    def on_failed(self, ignore):
        self.fails += 1

    def dowork1(self):
        ds = [ defer.succeed(1), defer.fail(ValueError('woops')) ]
        return defer.gatherResults(ds).addErrback(self.on_failed)

    def dowork2(self):
        return defer.fail(ValueError('woops')).addErrback(self.on_failed)

    def dowork3(self):
        return defer.fail(ValueError('woops')).addErrback(log.err).addErrback(self.on_failed)

class TheTestCase(TestCase):

    def test_failures1(self):
        component = SomeComponent()

        def check(result):
            self.assertEquals(component.fails, 1)
            print 'check succeeded'

        return component.dowork1().addCallback(check)

    def test_failures2(self):
        component = SomeComponent()

        def check(result):
            self.assertEquals(component.fails, 1)
            print 'check succeeded'

        return component.dowork2().addCallback(check)
    
    def test_failures3(self):
        component = SomeComponent()

        def check(result):
            self.assertEquals(component.fails, 0)
            print 'check succeeded'

        return component.dowork3().addCallback(check)

Output of test run on my machine:

test_wth
  TheTestCase
    test_failures1 ... check succeeded
                                                 [ERROR]
    test_failures2 ... check succeeded
                                                    [OK]
    test_failures3 ... check succeeded
                                                 [ERROR]

===============================================================================
[ERROR]: test_wth.TheTestCase.test_failures1

Traceback (most recent call last):
Failure: exceptions.ValueError: woops
===============================================================================
[ERROR]: test_wth.TheTestCase.test_failures3

Traceback (most recent call last):
Failure: exceptions.ValueError: woops
-------------------------------------------------------------------------------
Ran 3 tests in 0.005s

FAILED (errors=2, successes=1)

[Jean-Paul Calderone]

trial is working fine.

gatherResults doesn't pass consumeErrors to DeferredList. So the error in test_failures1 is not handled.

Logged errors are errors and trial fails tests when it sees them. Use TestCase.flushLoggedErrors if you want the test to pass.