Trial reports SUCCESS!?! and ERROR for the same test

[jknight]

In the failing test I'm just about to commit: 
test_internet.InterfaceTestCase.testCallLaterOrder2,
setting .todo doesn't actually work. 

Trial reports:
Running 1 tests.
  InterfaceTestCase
    testCallLaterOrder2 ...                                        [SUCCESS!?!]
                                            [ERROR]

===============================================================================
[ERROR]: twisted.test.test_internet.InterfaceTestCase.testCallLaterOrder2

Failure: twisted.trial.unittest.FailTest: [0, 2, 6, 12] != [0, 1, 2, 3, 4, 5, 6, 
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
===============================================================================
[SUCCESS!?!]: twisted.test.test_internet.InterfaceTestCase.testCallLaterOrder2

Reason: 'See bug 1396'

-------------------------------------------------------------------------------
Ran 1 tests in 0.557s

[Jonathan Lange]

testCallLaterOrder2 is managing to call log.err(). I guess this is because of
unhandled errors in callLaters.

So, this can be reduced to an earlier problem -- #1384

[Jonathan Lange]

My last comment is mistaken.

The real problem is that log error calls are handled in cleanup, and todo has no authority there.

This creates two topics of interest:

# Should .todo cover the entire test, or just the test method? # log.err calls should be handled differently.

#1409 addresses the second point.

What about the first? (CC: opening up the discussion to other vocal trial users)

[jknight]

Todo should cover the entire test.

[spiv]

I think I agree with jknight that todo should cover the entire test.

In general, it's surprising any time multiple results are associated with a single test.

[Jonathan Lange]

So, I'm happy now to make .todo cover the entire test.

However, as stated elsewhere (citation needed) I am committed to tests being able to have multiple results. Or rather, I am convinced that trial shouldn't hide errors, unless explicitly asked to (ala todo).