Opened 12 years ago

Closed 12 years ago

#1213 defect closed duplicate (duplicate)

trial does not isolate test methods from each other

Reported by: Jean-Paul Calderone Owned by:
Priority: high Milestone:
Component: trial Keywords: intermittent
Cc: Jean-Paul Calderone, spiv, Jonathan Lange, radix, Glyph Branch:
Author:

Description (last modified by Jean-Paul Calderone)

Attached is a failed run from the reactors buildslave.

The test which failed is twisted.mail.test.test_smtp.TimeoutTestCase.testSMTPClient, with the error being reported during twisted.lore.test.test_lore.TestFactory.test_doFile_withFilenameGenerator.

Attachments (1)

trialfailure.tar.gz (105.1 KB) - added by Jean-Paul Calderone 12 years ago.
Buildbot logs from bleed-over failure run

Download all attachments as: .zip

Change History (21)

comment:1 Changed 12 years ago by Jean-Paul Calderone

For example, it sometimes says things like this:

===============================================================================
[ERROR]: twisted.test.test_failure.FailureTestCase.testStringExceptions

  File "e:\buildslave\w32\W32-full2.2\Twisted\twisted\internet\defer.py", line
358, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "e:\buildslave\w32\W32-full2.2\Twisted\twisted\mail\test\test_imap.py",
line 822, in _ebGeneral
    self.client.transport.loseConnection()
exceptions.AttributeError: 'TLSTestCase' object has no attribute 'client'
===============================================================================

comment:2 Changed 12 years ago by Jonathan Lange

Can you please elaborate on what you mean by "trial does not isolate test
methods from each other"?

comment:3 Changed 12 years ago by Jean-Paul Calderone

An exception in test_imap is failing a test in test_failure.  This should not be
possible.

comment:4 Changed 12 years ago by Jonathan Lange

Hurrah for asynchronous programming!

comment:5 Changed 12 years ago by Jonathan Lange

I'm trying to figure out how this was possible so I know what to fix.

I guess SimpleClient.serverGreeting somehow got called after the IMAP tests were
finished -- but how?  There would have been errors raised in the logs if there
were pending calls, and I can't see any traces of threads, or invalid
reactor.iterate loops.  Maybe I'm not looking hard enough.

Incidentally, this sort of behaviour almost certainly doesn't happen if your
test just returns a Deferred.

comment:6 Changed 12 years ago by Jean-Paul Calderone

There should not be open, unassigned tickets in the tracker.

comment:7 Changed 12 years ago by Jonathan Lange

Fair call.

But I cannot reproduce this.  I do not want to have issues assigned to me that I
cannot act on.

comment:8 Changed 12 years ago by Jean-Paul Calderone

More fodder:

===============================================================================
[FAIL]: twisted.mail.test.test_smtp.TimeoutTestCase.testESMTPClient

  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/mail/test/test_smtp.py",
line 519, in testESMTPClient
    return self._timeoutTest(onDone, clientFactory)
  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/mail/test/test_smtp.py",
line 498, in _timeoutTest
    self.failIf(after - before > 1.0)
twisted.trial.unittest.FailTest: None
===============================================================================
[FAIL]:
twisted.trial.test.test_assertions.TestAssertions.test_failingException_fails

  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/trial/test/test_assertions.py",
line 49, in test_failingException_fails
    self.failUnlessEqual(result.errors, [])
twisted.trial.unittest.FailTest:
[(<twisted.trial.test.test_assertions.FailingTest testMethod=test_fails>,
<twisted.python.failure.Failure twisted.mail.smtp.SMTPTimeoutError>)] != []
-------------------------------------------------------------------------------

comment:9 Changed 12 years ago by Jean-Paul Calderone

This happend again on the gtk2 slave:

===============================================================================
[FAIL]: twisted.mail.test.test_smtp.TimeoutTestCase.testSMTPClient

  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/mail/test/test_smtp.py",
line 509, in testSMTPClient
    return self._timeoutTest(onDone, clientFactory)
  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/mail/test/test_smtp.py",
line 498, in _timeoutTest
    self.failIf(after - before > 1.0)
twisted.trial.unittest.FailTest: None
===============================================================================
[FAIL]:
twisted.trial.test.test_assertions.TestAssertions.test_failingException_fails

  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/trial/test/test_assertions.py",
line 49, in test_failingException_fails
    self.failUnlessEqual(result.errors, [])
twisted.trial.unittest.FailTest:
[(<twisted.trial.test.test_assertions.FailingTest testMethod=test_fails>,
<twisted.python.failure.Failure twisted.mail.smtp.SMTPTimeoutError>)] != []
-------------------------------------------------------------------------------

comment:10 Changed 12 years ago by Jean-Paul Calderone

Another example, from the Qt reactor:

===============================================================================
[FAIL]: twisted.mail.test.test_smtp.TimeoutTestCase.testSMTPClient

  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/mail/test/test_smtp.py",
line 509, in testSMTPClient
    return self._timeoutTest(onDone, clientFactory)
  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/mail/test/test_smtp.py",
line 498, in _timeoutTest
    self.failIf(after - before > 1.0)
twisted.trial.unittest.FailTest: None
===============================================================================
[ERROR]: twisted.trial.test.test_output.TestImportErrors.test_badmodule

Failure: twisted.mail.smtp.SMTPTimeoutError: <SMTPTimeoutError instance at
-1345903540 with str error Traceback (most recent call last):

  File
"/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/python/reflect.py", line
463, in safe_str
    return str(o)

  File "/home/buildbot/BuildBot/slave/reactors/Twisted/twisted/mail/smtp.py",
line 167, in __str__
    return '\n'.join(res)

TypeError: sequence item 2: expected string, instance found
>
-------------------------------------------------------------------------------

comment:11 Changed 12 years ago by Jonathan Lange

Component: coretrial

comment:12 Changed 12 years ago by Jonathan Lange

Cc: radix Glyph added

I still don't know what to do about this one. It clearly needs to be fixed.

Some thoughts:

  • It's probably a race.
  • All the errors seem to have something to do with twisted.mail -- don't know what this means
  • To even begin to reproduce, I'm going to have to examine all the logs of a buildbot when it happens
  • The comment dated "02/08/06 14:58:13" is interesting. It seems that somehow the smtp error is getting attached to a result object that we are creating within a test.

CCing more people because this problem needs more brain.

Next time it happens, can whoever notices please attach the buildbot logs to this issue, and note the non-isolated error in the comments? Thanks.

comment:13 Changed 12 years ago by Jonathan Lange

Keywords: intermittent added

comment:14 Changed 12 years ago by Jonathan Lange

Keywords: waiting added

Changed 12 years ago by Jean-Paul Calderone

Attachment: trialfailure.tar.gz added

Buildbot logs from bleed-over failure run

comment:15 Changed 12 years ago by Jean-Paul Calderone

Description: modified (diff)

comment:16 Changed 12 years ago by Jean-Paul Calderone

Keywords: waiting removed

comment:17 Changed 12 years ago by Jonathan Lange

Status: newassigned

It really happens so often? *sigh*

Thanks.

comment:18 Changed 12 years ago by Jonathan Lange

A fix to #1507 will probably fix this. The error seems to be caused by unhandled Deferreds being garbage collected at weird times.

comment:19 Changed 12 years ago by Jonathan Lange

Resolution: duplicate
Status: assignedclosed

comment:20 Changed 7 years ago by <automation>

Owner: Jonathan Lange deleted
Note: See TracTickets for help on using tickets.