[Twisted-Python] Trial & Hypothesis

[Jean-Paul Calderone]

Hey all,

A few folks had an IRC discussion about improving the trial & hypothesis
experience.  I took some notes and thought I'd share them here in case
anyone else is interested.

We're planning to pick up the discussion next Wednesday.  Feel free to join
in #twisted-dev on Freenode (we might set up a Google Hangout, details will
be on IRC if so).

Jean-Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://twistedmatrix.com/pipermail/twisted-python/attachments/20170215/994b7e39/attachment.html>
-------------- next part --------------
* Goals
** Make Twisted / Hypothesis as convenient as Anything Else / Hypothesis.

*** trial doesn't discover "runTest" so Hypothesis stateful tests aren't discovery
**** Make trial discover "runTest"
***** This might break things
***** Also TestLoader.getTestCaseNames is incompatible with test methods without a test_ prefix (so we need a new interface)

**** Make Hypothesis name its test "test_something"
***** Name could be an argument to a stateful TestCase factory

*** trial TestCases have extra helper methods not availabe to Hypothesis stateful tests
**** Make base TestCase class an argument to a stateful TestCase factory

*** setUp / tearDown are called just once even though test body might be called many times; and
*** tests that return Deferred are simply incompatible with Hypothesis
**** Hypothesis has an "executor" hook (eg new_style_executor) that might be a good place to hang reactor knowledge
**** this could also be a place to hang better setUp / tearDown handling
***** this would be incompatible with current operation which has separate setup_example/teardown_example hooks
***** and there's still only a single TestCase instance having methods called on it repeatedly which could be surprising
****** `trial -u` does this already though, so maybe less so for trial users
***** but it might be better, anyway

*** Hypothesis does a lot of work within a single test method.
**** A consequence is a ton of log data mixed together from runs with different values.
***** It would be nice to be able to see minimal logging from just the most shrunk iteration.
**** Maybe there could be a custom HypothesisSuite that knows how repeatedly instantiate a TestCase and run it more normally
***** This replaces the loop inside a test_ method with a loop around something more like TestCase().run()
****** might solve many problems at once
****** confuses things like `trial -j`
****** causes each test run to have a different number of tests, which is weird