Opened 9 years ago

Last modified 8 years ago

#1397 defect new

Trial reports SUCCESS!?! and ERROR for the same test

Reported by: jknight Owned by:
Priority: high Milestone:
Component: trial Keywords: waiting
Cc: jml, jknight, exarkun, spiv Branch:
Author: Launchpad Bug:

Description


Change History (8)

comment:1 Changed 9 years ago by jknight

In the failing test I'm just about to commit: 
test_internet.InterfaceTestCase.testCallLaterOrder2,
setting .todo doesn't actually work. 

Trial reports:
Running 1 tests.
  InterfaceTestCase
    testCallLaterOrder2 ...                                        [SUCCESS!?!]
                                            [ERROR]

===============================================================================
[ERROR]: twisted.test.test_internet.InterfaceTestCase.testCallLaterOrder2

Failure: twisted.trial.unittest.FailTest: [0, 2, 6, 12] != [0, 1, 2, 3, 4, 5, 6, 
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
===============================================================================
[SUCCESS!?!]: twisted.test.test_internet.InterfaceTestCase.testCallLaterOrder2

Reason: 'See bug 1396'

-------------------------------------------------------------------------------
Ran 1 tests in 0.557s

comment:2 Changed 9 years ago by jml

testCallLaterOrder2 is managing to call log.err(). I guess this is because of
unhandled errors in callLaters.

So, this can be reduced to an earlier problem -- #1384

comment:3 Changed 8 years ago by jml

  • Cc exarkun spiv added

My last comment is mistaken.

The real problem is that log error calls are handled in cleanup, and todo has no authority there.

This creates two topics of interest:

# Should .todo cover the entire test, or just the test method?
# log.err calls should be handled differently.

#1409 addresses the second point.

What about the first? (CC: opening up the discussion to other vocal trial users)

comment:4 Changed 8 years ago by jml

  • Keywords waiting added

comment:5 Changed 8 years ago by jknight

Todo should cover the entire test.

comment:6 Changed 8 years ago by spiv

I think I agree with jknight that todo should cover the entire test.

In general, it's surprising any time multiple results are associated with a single test.

comment:7 Changed 8 years ago by jml

So, I'm happy now to make .todo cover the entire test.

However, as stated elsewhere (citation needed) I am committed to tests being able to have multiple results. Or rather, I am convinced that trial shouldn't hide errors, unless explicitly asked to (ala todo).

comment:8 Changed 4 years ago by <automation>

  • Owner jml deleted
Note: See TracTickets for help on using tickets.