<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Jul 26, 2013, at 7:12 AM, <a href="mailto:exarkun@twistedmatrix.com">exarkun@twistedmatrix.com</a> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">To address this problem, I suggest you get into the habit of watching your unit tests fail in the expected way before you make the necessary implementation changes to make them pass.</span><br style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">This is only one of an unlimited number of ways your unit tests can be buggy. It might be tempting to try to fix the test runner to prevent you from ever falling into this trap again - and who knows, it might even be a good idea.</span><br style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">However, if you run your tests and see them fail in the way you expected them to fail before you write the code that makes them pass, then you will be sure to avoid the many, many, many *other* pitfalls that have nothing to do with accidentally returning the wrong object.</span><br style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">This is just one of the attractions of test-driven development for me.</span><br style="font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></blockquote></div><br><div>On a more serious note than our previous digression, perhaps *this* is the thing we should be modifying Trial to support.</div><div><br></div><div>The vast majority of Twisted committers do development this way - or at least aspire to, most of the time - but to someone new to automated testing, it's not entirely clear <i>how</i> you're supposed to use something like Trial, or how important it is that you see the tests fail first.</div><div><br></div><div>Perhaps if trial had a bit more of a memory of things that happened between test runs it would be useful. For example, a mode where you could tell it what you're working on, and you could just re-run the same thing and you'd only get a 'success' when you went back and forth between red and green.</div><div><br></div><div>Here's a silly little narrative about how one might use such a thing:</div><div><br></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div>$ tribulation begin myproject</div><div>Beginning a time of turmoil for python package 'myproject', in './myproject/'.</div><div>myproject.test_1</div><div> Case1</div><div> test_1 ... [OK]</div><div><br></div><div>-------------------------------------------------------------------------------</div><div>Ran 2 tests in 0.033s</div><div><br></div><div><div><font color="#009f49">PROCEED</font> (successes=1) - All tests passing, an auspicious beginning. Now write a failing test.</div><div>$ tribulation continue</div><div><div><div>myproject.test_1</div><div> Case1</div><div> test_1 ... [OK]</div><div>myproject.test_2</div><div> Case2</div><div> test_2 ... [OK]</div><div><br></div><div>-------------------------------------------------------------------------------</div><div>Ran 2 tests in 0.033s</div></div><div><br></div></div><div></div></div><div><font color="#a51400"><b>AGAIN</b></font> (successes=2) - a test should have failed.</div><div># oops, 'test_2' was just 'pass'... let me fix that</div><div>$ tribulation continue</div><div><div>$ tribulation begin myproject</div></div><div><div>Beginning a time of turmoil for python package 'myproject', in './myproject/'.</div></div><div><div>myproject.test_1</div></div><div><div> Case1</div></div><div><div> test_1 ... [OK]</div></div><div><div>myproject.test_2</div></div><div><div> Case2</div></div><div><div> test_2 ... <font color="#a51400">[FAIL]</font></div></div><div><div><br></div></div><div><div>-------------------------------------------------------------------------------</div></div><div><div>Ran 2 tests in 0.450s</div></div><div><div><br></div></div><div><div><font color="#009f49">PROCEED</font> (successes=2) - we are working on myproject.Case2.test_2 now.</div></div><div><div>$ tribulation continue</div></div><div><div>myproject.test_2</div></div><div><div> Case2</div></div><div><div> test_2 ... <font color="#a51400">[FAIL]</font></div></div><div><div><br></div></div><div><div>-------------------------------------------------------------------------------</div></div><div><div>Ran 1 tests in 0.020s</div></div><div><div><font color="#a51400"><b>AGAIN</b></font> (successes=2) - you should have made the test pass.</div></div><div>$ tribulation continue</div><div><div>myproject.test_2</div><div> Case2</div><div> test_2 ... [OK]</div><div><br></div><div>-------------------------------------------------------------------------------</div><div>Ran 1 tests in 0.01s</div></div><div><div><font color="#009f49">PROCEED</font> (successes=1) - myproject.Case2.test_2 works now, let's make sure nothing else broke.</div><div>$ tribulation continue</div><div><div>myproject.test_1</div><div> Case1</div><div> test_1 ... [OK]</div><div>myproject.test_2</div><div> Case2</div><div> test_2 ... [OK]</div><div><br></div><div>-------------------------------------------------------------------------------</div><div>Ran 2 tests in 0.033s</div></div><div><div><font color="#009f49">PROCEED</font> (successes=2) - no regressions, find the next thing to work on</div></div><div>$ tribulation conclude</div><div>You have received one billion points, congratulations you have defeated software.</div><div></div></div></blockquote><div><br></div><div>Does this seem like it might be a useful feature for someone to work on? Not shown here is the part that when you do introduce a regression, it runs just the tests that failed until you fix all of them, then goes back up the suite until it reaches the top and you move on to the next thing...</div><div><br></div><div>-glyph</div><div><br></div></body></html>