Ticket #3554 defect new

Opened 4 years ago

Last modified 4 years ago

Tests should not fail when .pycs are out of date or invalid

Reported by: radix Owned by:
Priority: normal Milestone:
Component: core Keywords:
Cc: exarkun Branch:
Author: Launchpad Bug:

Description

We have a number of tests which fail if .pycs are either out of date or point to non-existent .py files. One might say that this is a bug in Python, but it's a bug that the Python developers are certainly not going to fix, so it's something we have to deal with.

Making trial delete .pycs might be an option, and would probably solve the majority of cases where these tests fail. It would still fail if you have broken .pyc files in a non-writable directory, but that probably happens only very rarely.

Another option would be to change all of the tests so that they avoid the problem: some of them are asserting things about the formatted traceback, which could be avoided perhaps by invalidating a linecache or only asserting those things about freshly-generated .py files?

The warning testing infrastructure seems very sensitive to this issue, probably because it's checking filenames from something that honors the paths found in .pyc files, but I'm not sure.

Change History

1

Changed 4 years ago by glyph

#3552 spawned this ticket.

It strikes me that this might be overgeneral and we should focus first on the issue in question there, regarding flushWarnings?

2

Changed 4 years ago by exarkun

  • cc exarkun added

Maybe it is appropriately general, but there should be some sub-tickets for addressing specific related issues?

Aside from that, before fixing anything, we need a buildslave which covers the functionality we want.

Personally, I do want the flushWarnings issue fixed, since unfortunately it causes lots of tests to fail in the normal Combinator-based workflow (since Combinator copies pycs around, invaliding their filenames).

I'm not sure which other issues exist. I guess it's easy to find out.

[two minutes later]

Here are the failures if pycs are wrong: 

===============================================================================
[FAIL]: twisted.test.test_adbapi.DeprecationTestCase.test_safe

Traceback (most recent call last):
  File "twisted/test/test_adbapi.py", line 581, in test_safe
    safe, "test'")
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <function safe at 0x8d8d3e4> is not deprecated.
===============================================================================
[FAIL]: twisted.test.test_enterprise.QuotingTestCase.testQuoting

Traceback (most recent call last):
  File "twisted/test/test_enterprise.py", line 24, in testQuoting
    typ),
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <function quote at 0x902d764> is not deprecated.
===============================================================================
[FAIL]: twisted.test.test_enterprise.QuotingTestCase.test_getKeyColumnDeprecation

Traceback (most recent call last):
  File "twisted/test/test_enterprise.py", line 41, in test_getKeyColumnDeprecation
    self.callDeprecated(util._deprecatedVersion, util.getKeyColumn, Row, "foo")
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <function getKeyColumn at 0x902d6bc> is not deprecated.
===============================================================================
[FAIL]: twisted.test.test_enterprise.QuotingTestCase.test_safeDeprecation

Traceback (most recent call last):
  File "twisted/test/test_enterprise.py", line 32, in test_safeDeprecation
    self.callDeprecated(util._deprecatedVersion, util.safe, "foo")
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <function safe at 0x902d7d4> is not deprecated.
===============================================================================
[FAIL]: twisted.test.generator_failure_tests.TwoPointFiveFailureTests.test_inlineCallbacksTracebacks

Traceback (most recent call last):
  File "twisted/test/generator_failure_tests.py", line 45, in test_inlineCallbacksTracebacks
    "1/0"
twisted.trial.unittest.FailTest: not equal:
a = None
b = '1/0'

===============================================================================
[FAIL]: twisted.test.generator_failure_tests.TwoPointFiveFailureTests.test_throwExceptionIntoGenerator

Traceback (most recent call last):
  File "twisted/test/generator_failure_tests.py", line 79, in test_throwExceptionIntoGenerator
    self.assertEquals(traceback.extract_tb(stuff[0][2])[-1][-1], "1/0")
twisted.trial.unittest.FailTest: not equal:
a = None
b = '1/0'

===============================================================================
[FAIL]: twisted.test.test_failure.FailureTestCase.testRaiseExceptionWithTB

Traceback (most recent call last):
  File "twisted/test/test_failure.py", line 87, in testRaiseExceptionWithTB
    self.assertEquals(innerline, '1/0')
twisted.trial.unittest.FailTest: not equal:
a = None
b = '1/0'

===============================================================================
[FAIL]: twisted.test.test_newcred.DeprecatedUtilTests.test_challenge

Traceback (most recent call last):
  File "twisted/test/test_newcred.py", line 69, in test_challenge
    self.assertEqual(len(warnings), 2)
twisted.trial.unittest.FailTest: not equal:
a = 0
b = 2

===============================================================================
[FAIL]: twisted.test.test_pb.DeprecationTests.test_noOperationDeprecated

Traceback (most recent call last):
  File "twisted/test/test_pb.py", line 1760, in test_noOperationDeprecated
    pb.noOperation, 1, 2, x=3, y=4)
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <function noOperation at 0x90dd87c> is not deprecated.
===============================================================================
[FAIL]: twisted.test.test_pb.DeprecationTests.test_printTraceback

Traceback (most recent call last):
  File "twisted/test/test_pb.py", line 1770, in test_printTraceback
    "printTraceback deprecation fake traceback value")
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <function printTraceback at 0x90dda74> is not deprecated.
===============================================================================
[FAIL]: twisted.test.test_twistd.DeprecationTests.test_initialLog

Traceback (most recent call last):
  File "twisted/test/test_twistd.py", line 1343, in test_initialLog
    self.callDeprecated(Version("Twisted", 8, 2, 0), app.initialLog)
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <function initialLog at 0x83b2aac> is not deprecated.
===============================================================================
[FAIL]: twisted.trial.test.test_assertions.TestCallDeprecated.test_callDeprecatedCallsFunction

Traceback (most recent call last):
  File "twisted/trial/test/test_assertions.py", line 683, in test_callDeprecatedCallsFunction
    result = self.callDeprecated(self.version, self.oldMethod, 'foo')
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <bound method TestCallDeprecated.oldMethod of <twisted.trial.test.test_assertions.TestCallDeprecated testMethod=test_callDeprecatedCallsFunction>> is not deprecated.
===============================================================================
[FAIL]: twisted.trial.test.test_assertions.TestCallDeprecated.test_callDeprecatedSuppressesWarning

Traceback (most recent call last):
  File "twisted/trial/test/test_assertions.py", line 674, in test_callDeprecatedSuppressesWarning
    self.callDeprecated(self.version, self.oldMethod, 'foo')
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <bound method TestCallDeprecated.oldMethod of <twisted.trial.test.test_assertions.TestCallDeprecated testMethod=test_callDeprecatedSuppressesWarning>> is not deprecated.
===============================================================================
[FAIL]: twisted.trial.test.test_assertions.TestCallDeprecated.test_failsWithIncorrectDeprecation

Traceback (most recent call last):
  File "twisted/trial/test/test_assertions.py", line 711, in test_failsWithIncorrectDeprecation
    self.assertIn(getVersionString(self.version), str(exception))
twisted.trial.unittest.FailTest: 'Twisted 8.0.0' not in '<bound method TestCallDeprecated.oldMethod of <twisted.trial.test.test_assertions.TestCallDeprecated testMethod=test_failsWithIncorrectDeprecation>> is not deprecated.'
===============================================================================
[FAIL]: twisted.trial.test.test_assertions.TestCallDeprecated.test_nestedDeprecation

Traceback (most recent call last):
  File "twisted/trial/test/test_assertions.py", line 729, in test_nestedDeprecation
    self.callDeprecated(differentVersion, nestedDeprecation, 24)
  File "twisted/trial/unittest.py", line 1101, in callDeprecated

twisted.trial.unittest.FailTest: <function nestedDeprecation at 0xc27daac> is not deprecated.
===============================================================================
[FAIL]: twisted.trial.test.test_reporter.FormatFailures.test_formatDefault

Traceback (most recent call last):
  File "twisted/trial/test/test_reporter.py", line 278, in test_formatDefault
    'RuntimeError: foo'], tb.splitlines())
  File "twisted/trial/test/test_reporter.py", line 62, in stringComparison
    % (line_number, exp.pattern, out))
twisted.trial.unittest.FailTest: Line 2: '^\\s*$' did not match string '    def idle():'
===============================================================================
[FAIL]: twisted.trial.test.test_reporter.TestErrorReporting.testFormatErroredMethod

Traceback (most recent call last):
  File "twisted/trial/test/test_reporter.py", line 128, in testFormatErroredMethod
    self.stringComparison(match, output)
  File "twisted/trial/test/test_reporter.py", line 62, in stringComparison
    % (line_number, exp.pattern, out))
twisted.trial.unittest.FailTest: Line 4: '^\\s+raise FoolishError, .I am a broken setUp method.$' did not match string '    '
===============================================================================
[FAIL]: twisted.trial.test.test_reporter.TestErrorReporting.testFormatFailedMethod

Traceback (most recent call last):
  File "twisted/trial/test/test_reporter.py", line 142, in testFormatFailedMethod
    self.stringComparison(match, output)
  File "twisted/trial/test/test_reporter.py", line 62, in stringComparison
    % (line_number, exp.pattern, out))
twisted.trial.unittest.FailTest: Line 4: '^\\s+self\\.fail\\("I fail"\\)$' did not match string '    '
===============================================================================
[FAIL]: twisted.trial.test.test_reporter.TestErrorReporting.testHiddenException

Traceback (most recent call last):
  File "twisted/trial/test/test_reporter.py", line 193, in testHiddenException
    self.stringComparison(match, output)
  File "twisted/trial/test/test_reporter.py", line 62, in stringComparison
    % (line_number, exp.pattern, out))
twisted.trial.unittest.FailTest: Line 4: '^\\s+self\\.fail\\("Deliberate failure to mask the hidden exception"\\)$' did not match string '    '
===============================================================================
[FAIL]: twisted.trial.test.test_reporter.TestUncleanWarningWrapperErrorReporting.testFormatErroredMethod

Traceback (most recent call last):
  File "twisted/trial/test/test_reporter.py", line 128, in testFormatErroredMethod
    self.stringComparison(match, output)
  File "twisted/trial/test/test_reporter.py", line 62, in stringComparison
    % (line_number, exp.pattern, out))
twisted.trial.unittest.FailTest: Line 4: '^\\s+raise FoolishError, .I am a broken setUp method.$' did not match string '    '
===============================================================================
[FAIL]: twisted.trial.test.test_reporter.TestUncleanWarningWrapperErrorReporting.testFormatFailedMethod

Traceback (most recent call last):
  File "twisted/trial/test/test_reporter.py", line 142, in testFormatFailedMethod
    self.stringComparison(match, output)
  File "twisted/trial/test/test_reporter.py", line 62, in stringComparison
    % (line_number, exp.pattern, out))
twisted.trial.unittest.FailTest: Line 4: '^\\s+self\\.fail\\("I fail"\\)$' did not match string '    '
===============================================================================
[FAIL]: twisted.trial.test.test_reporter.TestUncleanWarningWrapperErrorReporting.testHiddenException

Traceback (most recent call last):
  File "twisted/trial/test/test_reporter.py", line 193, in testHiddenException
    self.stringComparison(match, output)
  File "twisted/trial/test/test_reporter.py", line 62, in stringComparison
    % (line_number, exp.pattern, out))
twisted.trial.unittest.FailTest: Line 4: '^\\s+self\\.fail\\("Deliberate failure to mask the hidden exception"\\)$' did not match string '    '
===============================================================================
[FAIL]: twisted.trial.test.test_warning.FlushWarningsTests.test_filterOnOffendingFunction

Traceback (most recent call last):
  File "twisted/trial/test/test_warning.py", line 241, in test_filterOnOffendingFunction
    [{'category': firstCategory, 'message': firstMessage}])
  File "twisted/trial/test/test_warning.py", line 64, in assertDictSubsets
    self.assertEqual(len(sets), len(subsets))
twisted.trial.unittest.FailTest: not equal:
a = 0
b = 1

===============================================================================
[ERROR]: twisted.test.test_newcred.DeprecatedUtilTests.test_respond

Traceback (most recent call last):
  File "twisted/test/test_newcred.py", line 48, in test_respond
    warnings[0]['message'],
exceptions.IndexError: list index out of range
===============================================================================
[ERROR]: twisted.trial.test.test_assertions.TestAssertions.test_assertWarnsAfterUnassertedWarning

Traceback (most recent call last):
  File "twisted/trial/test/test_assertions.py", line 538, in test_assertWarnsAfterUnassertedWarning
    [warning] = self.flushWarnings([f])
exceptions.ValueError: need more than 0 values to unpack
===============================================================================
[ERROR]: twisted.trial.test.test_tests.SuppressionTest.test_overrideSuppressClass

Traceback (most recent call last):
  File "twisted/trial/test/test_tests.py", line 578, in test_overrideSuppressClass
    warningsShown[0]['message'], suppression.METHOD_WARNING_MSG)
exceptions.IndexError: list index out of range
===============================================================================
[ERROR]: twisted.trial.test.test_tests.SuppressionTest.test_suppressClass

Traceback (most recent call last):
  File "twisted/trial/test/test_tests.py", line 543, in test_suppressClass
    warningsShown[0]['message'], suppression.METHOD_WARNING_MSG)
exceptions.IndexError: list index out of range
===============================================================================
[ERROR]: twisted.trial.test.test_tests.SuppressionTest.test_suppressMethod

Traceback (most recent call last):
  File "twisted/trial/test/test_tests.py", line 526, in test_suppressMethod
    warningsShown[0]['message'], suppression.CLASS_WARNING_MSG)
exceptions.IndexError: list index out of range
===============================================================================
[ERROR]: twisted.trial.test.test_tests.SuppressionTest.test_suppressModule

Traceback (most recent call last):
  File "twisted/trial/test/test_tests.py", line 560, in test_suppressModule
    warningsShown[0]['message'], suppression.METHOD_WARNING_MSG)
exceptions.IndexError: list index out of range
===============================================================================
[ERROR]: twisted.trial.test.test_warning.FlushWarningsTests.test_unflushed

Traceback (most recent call last):
  File "twisted/trial/test/test_warning.py", line 134, in test_unflushed
    self.assertEqual(warningsShown[0]['message'], 'some warning text')
exceptions.IndexError: list index out of range
-------------------------------------------------------------------------------

Most fail because of flushWarnings not working in this case. These, however, fail for some other reason:

  • twisted.test.generator_failure_tests.TwoPointFiveFailureTests.test_inlineCallbacksTracebacks
  • twisted.test.generator_failure_tests.TwoPointFiveFailureTests.test_throwExceptionIntoGenerator
  • twisted.test.test_failure.FailureTestCase.testRaiseExceptionWithTB
  • twisted.trial.test.test_reporter.FormatFailures.test_formatDefault
  • twisted.trial.test.test_reporter.TestErrorReporting.testFormatErroredMethod
  • twisted.trial.test.test_reporter.TestErrorReporting.testFormatFailedMethod
  • twisted.trial.test.test_reporter.TestErrorReporting.testHiddenException
  • twisted.trial.test.test_reporter.TestUncleanWarningWrapperErrorReporting.testFormatErroredMethod
  • twisted.trial.test.test_reporter.TestUncleanWarningWrapperErrorReporting.testFormatFailedMethod
  • twisted.trial.test.test_reporter.TestUncleanWarningWrapperErrorReporting.testHiddenException

So perhaps there should be three more specific tickets. One for flushWarnings, one for the Failure issues, one for the trial reporter issues. Four tickets. One for a buildbot configuration.

3

Changed 4 years ago by exarkun

I've created #3598 for the flushWarnings issue.

4

Changed 4 years ago by glyph

  • owner changed from glyph to radix

5

Changed 2 years ago by <automation>

  • owner radix deleted
Note: See TracTickets for help on using tickets.