[Twisted-Python] Refactoring of trial - call for feature requests and suggestions

[Cory Dodt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Glyph Lefkowitz wrote:
[...]
| Second is that we shouldn't be waiting on deferreds in test code any
| more than we should be waiting on deferreds in real code.  The different
| approaches to waiting on deferreds currently mean there are still lots
| of places the tests can hang.  With a more unified approach, we could
| insert default timeouts on ALL tests more easily, and avoid loops that
| wait for multiple deferreds by using, e.g. DeferredList.

I would be a big fan of timeouts on all tests.  There are several different
ways win32 tests can hang.  We can't even run the win32ER tests currently, due
to hangs.  In my recent testing of wprocess (processes for windows in default
reactor) I decided to just blindly enable all the process tests and see which
ones broke.  I didn't find out though, because they hang in many different places.


On an unrelated note..

What if we had persistent test results?  Perhaps a test result aggregator
running on pyr.  Such a thing would enable us (independently of buildbot) to
examine trends and pull out tests that frequently regress--indicating either
(1) very fragile code that should be replaced or (2) more likely, tests that
fail only intermittently; and tests that have been broken (but are
expectedfailures or whatever) for a long time.

It doesn't have to be very complex.  Just something that accepts and parses
the test runs in sort of the same way that buildbot does, but can identify a
particular test from the log and record whether it passed or failed, and
compare several runs of that one test (or groups of tests, grouped in whatever
way makes the most sense) over time.  Maybe a command-line tool that would let
someone on pyr do the obvious analyses.  (Or a web tool, if someone is really
just dying to write one.)

The buildmaster would just have to pipe test runs into it.

C


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3-nr1 (Windows XP)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFAz29d3A5SrXAiHQcRApB5AJ9fJ3ZY7j4neTPB5QtKgbJSlJUGagCgqkl+
lI7Exzkcl91ZsUArA/nxYOw=
=zA8M
-----END PGP SIGNATURE-----