Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#6336 enhancement closed wontfix (wontfix)

twisted.trial.unittest does not support Python unittest's @unittest.expectedFailure

Reported by: I. Dleaney Owned by:
Priority: normal Milestone:
Component: trial Keywords:
Cc: Jonathan Lange Branch:
Author:

Description (last modified by Glyph)

version nose-1.2.1, nose-1.2.1_p2012.

nose-1.2.1/functional_tests/support/twist/test_twisted.py;

from twisted.trial import unittest

class TestTwisted(unittest.TestCase):

    def test(self):
        pass

    def test_fail(self):
        self.fail("I failed")

    def test_error(self):
        raise TypeError("oops, wrong type")

    def test_skip(self):
        raise unittest.SkipTest('skip me')

This is sooooo annoying, and I can't tell who' more in the wrong; nose for adding a test to their suite that is a compulsory expected fail and offering no facility to mark it an expected fail, or twisted for not adding expectedFailure to twisted.trial.unittest. Toss a coin. Either way, the test is a compulsory fail and sets running of the suite to exit with fail or error on no.s 2 & 3.

I tried setting

@unittest.expectedFailure for

def test_fail(self):

and

def test_error(self):

and it comes up no such attribute. Please add it.

Change History (13)

comment:1 Changed 7 years ago by DefaultCC Plugin

Cc: Jonathan Lange added

comment:2 Changed 7 years ago by therve

Milestone: Twisted-12.3
Resolution: invalid
Status: newclosed

The bug doesn't make much sense to me. It seem you want us to fix nose test suite? Please formulate something more clear, or send something to the mailing list. Thanks.

comment:3 Changed 7 years ago by Glyph

Description: modified (diff)

(formatting fix, please ignore)

comment:4 Changed 7 years ago by I. Dleaney

It seem you want us to fix nose test suite?

No, that's silly. Don't worry I filed it there too.

nose test suite employs twisted's trial via

from twisted.trial import unittest

in

nose-1.2.1/functional_tests/support/twist/test_twisted.py

What doesn't make much sense is setting a test file to fail, not pass. In this respect, you're the innocent party.

unittest under py2.7 offers the decorator or facility

@unittest.expectedFailure

which once used would make the test return success because it failed.

Unless I'm wrong the trial.unittest under twisted appears to not have this decorator included. I was asking if you could add it, making the test return a sane result.

status changed from new to closed resolution set to invalid milestone Twisted-12.3 deleted

you said no. I say I tried

comment:5 Changed 7 years ago by therve

Trial already supports expected failures via the todo mechanism.

comment:6 in reply to:  5 Changed 7 years ago by Glyph

Replying to therve:

Trial already supports expected failures via the todo mechanism.

Where do we document this mechanism? I was looking to post a link to make it clearer, but perhaps we have a doc bug: I don't see it documented either in Writing tests for Twisted code using Trial or Test-driven development with Twisted.

comment:8 in reply to:  7 Changed 7 years ago by I. Dleaney

Replying to therve:

It's documented a bit indirectly here: https://twistedmatrix.com/documents/current/core/development/policy/test-standard.html#auto10

therve,

ok hey thx for reply. "Trial already supports expected failures via the todo mechanism."

roger that.. Well, I didn't peruse twisted docs, I did just jump straight to the py2.7 from of @unittest.expectedFailure decorator, and once that didn't work as such I then concluded it's not there.

However, this IS a nose test suite test that's doing a test by use of twisted's trial which imports twisted.trial.unittest and not cpython's unittest. Once again, on yr behalf, twisted is the innocent party here. I did file at nose also and you have replied nice and promptly. Thanks for that. (I'm not quite as annoyed and grumpy as I was when I filed!!) By rights the nose test suite writers could have should have utilised it and made the test return a success on failure rather than a failure on failure.

comment:9 Changed 7 years ago by Itamar Turner-Trauring

Resolution: invalid
Status: closedreopened
Summary: twisted.trial.unittest runs nose's testsuite into failuretwisted.trial.unittest does not support Python unittest's @unittest.expectedFailure

In general we should support all built-in features in Python's unittest module in trial as well; we're supposed to be extending it, not breaking it. I changed the summary to be a more accurate explanation of the problem. Idella, could you rewrite the ticket description with a shorter version explaining just the actual issue regarding expectedFailure?

comment:10 Changed 7 years ago by Jean-Paul Calderone

In general we should support all built-in features in Python's unittest module in trial as well

I think you're misunderstanding or misconstruing the goals for trial/stdlib interoperability. The primary promise that we made and should uphold is that trial can run unittest.TestCase subclasses.

comment:11 Changed 7 years ago by Jean-Paul Calderone

Resolution: wontfix
Status: reopenedclosed

To avoid the confusion of this ticket, please see #6345. Thanks, everyone.

comment:12 Changed 7 years ago by Itamar Turner-Trauring

That's a more accurate way of saying what I was aiming at. E.g. I don't think "have a crappy runner" is a goal just because Python's unittest has one.

comment:13 Changed 7 years ago by I. Dleaney

Re-write of the description

  1. nose test suite employs twisted's trial via
  2. from twisted.trial import unittest
  3. in nose-1.2.1/functional_tests/support/twist/test_twisted.py
  1. The 1 line tests written by those at nose were self.fail("I failed") and raise TypeError("oops, wrong type")

making for intended compulsory 1 fail 1 error.

What made these calls kind of bullet proof was that the final file that called it was a system installed file of twisted. Not being an insource file, I had no option of editing or patching that compulsory call.

  1. What I attempted to do was to decorate these 2 calls to fail with Cpython's style of @unittest.expectedFailure since that's really what they are.
  1. The testsuite of nose had called the tests from another insource file, which added to the dilemma by putting to console the reporting of the effective 1 fail 1 error. To this test I could actually invoke CPython's unittest.skip, fixing the issue of compulsory failure.
  1. Having filed it also with upstream nose, happily a maintainer from there has acknowledged this compulsory call to 1 fail 1 error and has made a fix, only in the last 4 hours.

  1. While I see the TODO mechanism cited, it appears to me intended as a temporary marker of a test until it is further refined and a cause resolved, making it not quite the substitute of expectedFailure.
Note: See TracTickets for help on using tickets.