Opened 2 years ago

Closed 13 months ago

#5787 enhancement closed fixed (fixed)

trial --order for selecting an order for running test cases

Reported by: Julian Owned by: Julian
Priority: normal Milestone:
Component: trial Keywords:
Cc: jml, thijs Branch: branches/trial-order-5787-2
(diff, github, buildbot, log)
Author: julian, tomprince Launchpad Bug:

Description

Before you yell too loud, yes, I know test methods should be order independent. I also knowingly admit that as presented the patch is non-ideal, part of which due to me still not being familiar with all of trial's parts, so if someone's got a better idea for how to implement it I'm all ears.

That being said... when writing test cases I often happen to write them so that simpler tests are defined earlier in the test case. I therefore like to keep the test output in that order so that the tests that fail at the top are generally the ones that need addressing. Attached is a patch that attempts to implement that. In conjunction with #1518 which I really do hope to get to at some point though I keep putting it off, I think these two at least would help my personal workflow for fixing failing tests.

Open for criticism.

P.S., we could deprecate --random / -z and / or rename it to --order=random if that's something anyone would like.

Attachments (10)

toptobottom.patch (5.9 KB) - added by Julian 2 years ago.
updated_top_bottom_patch.patch (5.5 KB) - added by cyli 22 months ago.
toptobottom-take-2.patch (9.9 KB) - added by Julian 21 months ago.
toptobottom-3.patch (11.2 KB) - added by Julian 18 months ago.
toptobottom-4.patch (16.0 KB) - added by Julian 16 months ago.
toptobottom-5.patch (16.2 KB) - added by Julian 16 months ago.
usage-error.patch (7.1 KB) - added by Julian 16 months ago.
trial-order.patch (4.4 KB) - added by Julian 15 months ago.
trial-order.2.patch (2.0 KB) - added by Julian 15 months ago.
trial-order.3.patch (2.6 KB) - added by Julian 14 months ago.

Download all attachments as: .zip

Change History (53)

comment:1 Changed 2 years ago by DefaultCC Plugin

  • Cc jml added

Changed 2 years ago by Julian

comment:2 Changed 22 months ago by cyli

  • Keywords review removed
  • Owner set to Julian

Hi Julian!

Thank you so much for working on this! I also think it'd be useful to be able to run tests in the order they're written in certain cases. (I also noticed that you worked on #5520, so cool).

There are two major issues though. Firstly, the patch seems to be out of date with trunk. I tried to merge it manually, and I've included an updated patch that I think would work, but you might want to check it.

Second is that when you pass a file or a module, the test cases don't seem to be constructed with a _testMethodName.

For example:

$ trial --order=toptobottom twisted.trial.test.test_script
Traceback (most recent call last):
  File "/Users/cyli/Projects/twisted/trunk/bin/trial", line 18, in <module>
    run()
  File "/Users/cyli/Projects/Twisted/trunk/twisted/scripts/trial.py", line 506, in run
    suite = _getSuite(config)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/scripts/trial.py", line 435, in _getSuite
    return loader.loadByNames(config['tests'], recurse)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 655, in loadByNames
    for thing in self._uniqueTests(things)]
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 613, in loadAnything
    return self.loadModule(thing)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 485, in loadModule
    for testClass in self.findTestClasses(module):
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 447, in findTestClasses
    return self.sort(classes)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 439, in sort
    return sorted(xs, key=self.sorter)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/scripts/trial.py", line 439, in _maybeFindSourceLine
    method = getattr(test_case, test_case._testMethodName)
AttributeError: type object 'CoverageTests' has no attribute '_testMethodName'

Also, I think a module gets passed in if you give trial a module, so:

$ trial --order=toptobottom twisted
Traceback (most recent call last):
  File "/Users/cyli/Projects/twisted/trunk/bin/trial", line 18, in <module>
    run()
  File "/Users/cyli/Projects/Twisted/trunk/twisted/scripts/trial.py", line 506, in run
    suite = _getSuite(config)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/scripts/trial.py", line 435, in _getSuite
    return loader.loadByNames(config['tests'], recurse)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 655, in loadByNames
    for thing in self._uniqueTests(things)]
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 612, in loadAnything
    return self.loadPackage(thing, recurse)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 561, in loadPackage
    for modinfo in self.sort(discovered):
  File "/Users/cyli/Projects/Twisted/trunk/twisted/trial/runner.py", line 439, in sort
    return sorted(xs, key=self.sorter)
  File "/Users/cyli/Projects/Twisted/trunk/twisted/scripts/trial.py", line 439, in _maybeFindSourceLine
    method = getattr(test_case, test_case._testMethodName)
AttributeError: 'PythonModule' object has no attribute '_testMethodName'

Perhaps the method should check to see if it's actually a test case first? (see twisted.trial.runner.name)

These are more minor comments:

  1. Doesn't seem like the TestCases in a file are not ordered from top to bottom? Perhaps that should also be stated in the usage docs? They seem to be alphabetical, maybe, or hierarchical? Whatever the default is.
  1. There should be 3 lines between class definitions and 2 lines between class methods, as per the twisted coding standard

Once again, thanks for all your work on trial! It is greatly appreciated!

Changed 22 months ago by cyli

comment:3 Changed 22 months ago by cyli

Oops, one more - could you update the man page?

comment:4 Changed 22 months ago by cyli

< sorry, I keep forgetting things. Could you please also add a news file? (see http://twistedmatrix.com/trac/wiki/ReviewProcess#Newsfiles)

comment:5 Changed 21 months ago by Julian

  • Keywords review added
  • Owner Julian deleted
  • Priority changed from low to normal

Phew, sorry, this one took a bit longer to get done :).

Added another try that I hope addresses these adequately. Thanks again :).

Changed 21 months ago by Julian

comment:6 Changed 21 months ago by Julian

Running the full test suite under --toptobottom gives me 2 errors (in twisted.python.test.test_deprecate). But looking at them I think they're tests that rely on ordering, so I've filed that as 6150.

comment:7 Changed 21 months ago by exarkun

Something that would have been nice to see here is an extensible mechanism for selecting test run ordering. Twisted has a very power plugin mechanism and trial already uses it in a few places. This comment isn't meant as a rejection of this set of changes, but I hope that someone will consider improving this feature after it lands by allowing it to be extended with plugins.

comment:8 Changed 18 months ago by exarkun

  • Owner set to exarkun
  • Status changed from new to assigned

comment:9 Changed 18 months ago by exarkun

  • Keywords review removed
  • Owner changed from exarkun to Julian
  • Status changed from assigned to new

Thanks.

  1. _run_orders does not follow the naming convention. _runOrders is the preferred spelling.
  2. There's no test coverage for the compData change
  3. There's no test coverage for opt_help_orders.
  4. OrderTests.tearDown doesn't seme very useful. Generally there's no reason to clear attributes like this.
  5. test_toptobottom_module does not follow the naming convention. test_toptobottomModule is the preferred spelling.
  6. ordertests.py is missing a copyright statement and a module docstring
  7. The test method docstrings are a good start, but could be better in a few ways. For example, with --alphabetical runs test methods alphabetically.:
    1. There is no --alphabetical option
    2. The command line option itself is not what runs anything, it just supplies configuration to the part of trial that does run tests.
    3. How is alphabetical defined? Is it based on the method name? The fully qualified name?
  8. Also, the docstring for test_toptobottom_module seems needlessly inconsistent. Why does --toptobottom sort tests when the other option runs them?
  9. If possible, it would be better to touch fewer internals in the tests. The fewer implementation details used, the more robust the tests. test_toptobottom_module is implemented in terms of testNames. This seems like an easy way to avoid using _tests or _testMethodName in the other tests.

Re-implementing the randomization logic in terms of the --order option, then deprecating the --random option, seems fine - if you can see a way to support all of the current behavior of --random. Note that it accepts an argument which controls its behavior, which --order can't obviously support as-is (which might suggest a weakness in this particular interface that should be addressed before it is finalized). I suggest submitting the deprecation part of that work as a separate ticket, though.

comment:10 Changed 18 months ago by Julian

  • Keywords review added

Hi. Thanks for the review.

Updated patch hopefully addresses all this.

The only thing unaddressed is #2. There isn't any way to test interaction with the completion system, and in a quick discussion with teratorn on IRC, though I think I probably understood where you were coming from in expecting coverage for it (a test testing that --order properly put itself in compData), I had a hard time disagreeing that any test would essentially just be duplicating the valid completions a second time in the test code and couldn't narrow down a sufficiently specific thing to test to come to an agreement without re-testing what _shellcomp already tests. Perhaps you can elaborate if you feel strongly about it.

Also, for --random, yeah, I was planning on another ticket, and assuming that, in combination with your request for plugin integration, different orders would presumably use envvars to get themselves additional options.

comment:11 Changed 18 months ago by Julian

  • Owner Julian deleted

Changed 18 months ago by Julian

comment:12 Changed 18 months ago by tom.prince

  • Owner set to tom.prince

comment:13 Changed 18 months ago by tom.prince

  • Keywords review removed
  • Owner changed from tom.prince to Julian
  1. _maybeFindSourceLine currently depends on the source file existing, which is perhaps reasonable. However, I think it isn't required. SyncTestCase.flushWarnings uses dis.findlinestarts to get the same information, without requiring the sourcefile.
  2. The descriptions printed could be all mixed up, and yet test_help_ordersPrintsSynopsisAndQuits wouldn't notice.
  3. There should be a test that tests what happens if alphabetical is specified and a module is passed.
  4. Is there any reason for removing the blank line after the list of reporters? (and not having one for --help-orders)
  5. What happens if a package is passed, instead of a module (with both options)? There should probably be a test for this as well.
  6. I'll let exarkun comment on compData.

Please resubmit after addressing the above (and getting feedback from exarkun on compData testing).

comment:14 Changed 17 months ago by exarkun

I closed #1948 as a duplicate, partly of this ticket and partly of #5520.

comment:15 Changed 16 months ago by Julian

  • Keywords review added

Hey Tom.

I tightened 2 up a tiny bit, you're right that it could be garbled but I think doing so would be fairly noticeable to someone implementing.

Added tests for packages and #1.

I think the blank line was removed during cyli's py3 port, though I'm not sure it was intentional. Personally I don't really like / want trailing blank lines unless there's further text afterwards, so I'll leave it off unless you prefer otherwise.

Addressed all the rest too hopefully.

Changed 16 months ago by Julian

comment:16 Changed 16 months ago by Julian

  • Owner Julian deleted

comment:17 Changed 16 months ago by cyli

Was not intentional no. :) sorry

comment:18 Changed 16 months ago by thijs

  • Cc thijs added
  • Keywords review removed
  • Owner set to Julian
  1. doc/core/man/trial.1
    1. the copyright date at the bottom needs to be updated
  2. twisted/scripts/trial.py
    1. all stdlib imports should be on separate lines
    2. opt_order has blank lines and is missing epydoc info for order param
    3. _maybeFindSourceLine has blank lines. The param name thing should be more descriptive and documented using epydoc markup
  3. twisted/topfiles/5787.feature
    1. a --order should be an --order
  4. twisted/trial/test/ordertests.py
    1. FooTest's methods are missing docstrings
    2. same for BazTest and BarTest
    3. it looks like there's a copy/paste error duplicating the module content twice. if this is intended, add some comments why :)
  5. twisted/trial/test/test_script.py
    1. (existing) stdlib imports need to be on separate lines
    2. no need to have that whitespace above OrderTests.setUp
    3. same for that first blank line in methods in the rest of the tests

Thanks.

comment:19 Changed 16 months ago by Julian

  • Keywords review added
  • Owner Julian deleted

Addressed all those I hope.

Other than docstrings for methods on FooTest & co. These are value objects, there's nothing meaningful to say about the methods beyond the class' docstring, they're not meant to test anything and won't be run. I did add docstrings to BarTest and BazTest which were missing.

Changed 16 months ago by Julian

comment:20 Changed 16 months ago by tomprince

  • Author set to tomprince
  • Branch set to branches/trial-order-5787

(In [37907]) Branching to trial-order-5787.

comment:21 Changed 16 months ago by tomprince

(In [37908]) Apply toptobottom-5.patch from Julian.

Refs: #5787

comment:22 Changed 16 months ago by tom.prince

  • Keywords review removed
  • Owner set to Julian
  1. test_toptobottomPackage doesn't actually assert anything, since the key used in sorting is always twisted

`
W9208:223,4:_BasicOptions.opt_help_orders: Missing docstring
W9202:457,0:_maybeFindSourceLine: Missing epytext markup @param for argument "thing"
`

  1. Looking a the actual output of "bin/trial --help", and the output for '--order' is the docstring, not the expected output. In any case, I think the descriptions from the manpage would be appropriate for the --help output.

comment:23 Changed 16 months ago by Julian

  • Keywords review added
  • Owner Julian deleted

Addressed 1, nice catch.

The 'thing' twistedchecker output was actually a docbug, so that's nice. I moved to a different approach which sidesteps 3, because I like this better anyhow. I want "[default: alphabetical]" to show up in --help, and it does now. I'm leaving off the docstring for opt_help_orders since it seems like that's what opt_help_reporters does, presumably due to your point in 3, but if you prefer I can just give it a docstring that is the same as the one in optParameters. usage.Options is kinda wonky, but I see you filed #6319.

There also was a missing test case for an unknown --order, which I added along with this.

Incremental patch is usage-error.patch.

Changed 16 months ago by Julian

comment:24 Changed 15 months ago by rwall

  • Owner set to rwall
  • Status changed from new to assigned

reviewing...

comment:25 Changed 15 months ago by rwall

(In [38199]) apply usage-error.patch from Julian. refs #5787

comment:26 Changed 15 months ago by rwall

  • Keywords review removed
  • Owner changed from rwall to Julian
  • Status changed from assigned to new

Code Review

Hi Julian,

Here's a quick code review of your branch including the latest
usage-error.patch

Notes:

  • Overall the code looks really nice this seems like a useful feature. Took me a while to comprehend how _maybeFindSourceLine works, but I learned a lot about trial and unittest in the process. :)
  • Patch applied cleanly
  • Branch merges cleanly to trunk
  • Coverage seems complete: https://buildbot.twistedmatrix.com/builds/twisted-coverage.py/twisted-coverage.py-r38199/twisted_scripts_trial.html
  • Seems to work with distrial. I ran trial -j4 --order=toptobottom twisted and one test failed (twisted.python.test.test_deprecate.TestDeprecationWarnings.test_deprecatedReplacement) as mentioned above.

Please address or answer the following minor points.

  1. Build Failures
    1. documentation: https://buildbot.twistedmatrix.com/builders/documentation/builds/3250/steps/api-documentation/logs/new%20pydoctor%20errors
    2. twistedchecker: W9501:158,12:_coerceOrder: String formatting operations should always use a tuplefor non-mapping values
  1. source:branches/trial-order-5787/twisted/scripts/trial.py
    1. _maybeFindSourceLine
      1. It took me a long time to understand what types of testThings can be passed to this function and which of them are likely to have _testMethodName. Maybe add some further @param and @type documentation and some comments to explain things.
      2. Too many blank lines above @param
      3. Elsewhere in twisted I see that we use dis.findlinestarts(method.code). I think code is the newer name for the code object.
  1. opt_help_orders: ugly wrapping in --help-orders. remove the line break "different\n orders"
  2. _getLoader: Maybe raise a usage error when --random and --order are both provided.
  1. source:branches/trial-order-5787/twisted/trial/test/test_script.py
    1. HelpOrderTests:
      1. Could you use assertRaises(SystemExit... instead of patching sys.exit?
  1. Did you raise tickets for all the suggestions in previous code reviews? (trial order plugin, order=random:1, deprecate --random)

Submit another patch against source:branches/trial-order-5787 after
answering or addressing the numbered points above.

-RichardW.

comment:27 follow-up: Changed 15 months ago by Julian

  • Keywords review added
  • Owner Julian deleted

Hey. Thanks for the review Richard.

Addressed all these hopefully.

Re: 4, plugin is in #6450 and deprecating random is in #6451.

Changed 15 months ago by Julian

comment:28 Changed 15 months ago by rwall

(In [38207]) apply trial-order.patch from Julian. refs #5787

comment:29 in reply to: ↑ 27 Changed 15 months ago by rwall

Replying to Julian:

Hey. Thanks for the review Richard.

Addressed all these hopefully.

Re: 4, plugin is in #6450 and deprecating random is in #6451.

Thanks.

I applied your latest patch to the branch and started another build:

This looks branch now looks fine to me (assuming the builds succeed), but I think I'll leave it to someone with more experience of trial to do a final review and merge.

-RichardW.

comment:30 Changed 15 months ago by rwall

Ah actually there is quite a serious test failure:

[ERROR]
Traceback (most recent call last):
  File "/var/lib/buildbot/bot-glyph-1/lucid32-py2.7maint/Twisted/twisted/trial/test/test_script.py", line 476, in test_getWorkerArguments
    "4", "--disablegc"])
  File "/var/lib/buildbot/bot-glyph-1/lucid32-py2.7maint/Twisted/twisted/python/usage.py", line 276, in parseOptions
    self.postOptions()
  File "/var/lib/buildbot/bot-glyph-1/lucid32-py2.7maint/Twisted/twisted/scripts/trial.py", line 471, in postOptions
    "You can't specify --random when using --order")
twisted.python.usage.UsageError: You can't specify --random when using --order

twisted.trial.test.test_script.OptionsTestCase.test_getWorkerArguments

Did you run trial locally before submitting the patch?

comment:31 follow-up: Changed 15 months ago by Julian

I saw, think especially considering the pending deprecation, I'll just remove the test and put a note in the man page to not pass them together. usage.Options does not appear to have a reasonable way to declare two flags to be conflicting here.

Changed 15 months ago by Julian

comment:32 Changed 15 months ago by rwall

(In [38209]) Apply trial-order.2.patch from Julian. refs #5787

comment:33 in reply to: ↑ 31 Changed 15 months ago by rwall

Replying to Julian:

I saw, think especially considering the pending deprecation, I'll just remove the test and put a note in the man page to not pass them together. usage.Options does not appear to have a reasonable way to declare two flags to be conflicting here.

Ok fair enough.

I applied your last patch and started another build:

Keep an eye on it.

comment:34 Changed 14 months ago by tom.prince

  • Keywords review removed
  • Owner set to Julian
  1. This branch uses some test-data in the source. As suggested in #6535, it would be better if the tests created this data (in _trial_temp).
  2. I think it would be good to disallow --order and --random together, even if the later is eventually going to be deprecated. One way to handle this would be to make the default for --order to be None, and change that to the real default after checking for --random.
  3. _coerceOrder is confusingly named.
  4. The comments on #6427 suggest that we want to have docstrings on opt_* methods. Since that hasn't been implemented yet, can you leave a comment on the methods without docstrings, with a pointer to that ticket, and a comment on that ticket to re-add the docstrings, once it is resolved.

Other than that, it looks good. Thanks for your work on this.

comment:35 Changed 14 months ago by Julian

  • Keywords review added
  • Owner Julian deleted

Hi Tom, thanks for the review.

Here's a patch for 2, 3, and 4, all of which sound fine to me. For 1, I agree that the current way is bad, but I'd love to not make this the first thing that has to adhere to a new standard (especially one that isn't documented / standardized yet) so that I don't potentially get chased around as we figure out how best to place things on a ticket that's essentially done. So if it's alright with you I'd like to leave that as-is. I presume that at some point there will be tickets for cleaning up some of the test data since trial has plenty of it, and doing that in one swoop wouldn't be hard I don't think? If not, feel free to take this out of review for now until I can spend some time on that.

Changed 14 months ago by Julian

comment:36 Changed 13 months ago by rwall

  • Owner set to rwall
  • Status changed from new to assigned

Reviewing...

comment:37 Changed 13 months ago by rwall

  • Keywords review removed
  • Owner changed from rwall to Julian
  • Status changed from assigned to new

Thanks for the latest patch.

Notes:

Problems:

  1. Merge conflicts (a simple conflict with your exit-first tests I think). Please merge forward and force a build.
    [richard@zorin trial-order-5787]$ unbranch Twisted
    ~/projects/Twisted/trunk$ svn status --quiet
    C:
    ~/projects/Twisted/trunk$ svn up
    C: Updating '.':
    C: At revision 38958.
    ~/projects/Twisted/trunk$ svn merge --non-interactive /home/richard/projects/Twisted/branches/trial-order-5787/@37907 /home/richard/projects/Twisted/branches/trial-order-5787/@HEAD
    C: --- Merging r37908 through r38958 into '.':
    C: A    twisted/topfiles/5787.feature
    C: U    twisted/scripts/trial.py
    C: A    twisted/trial/test/ordertests.py
    C: C    twisted/trial/test/test_script.py
    C: U    doc/core/man/trial.1
    C: Summary of conflicts:
    C:   Text conflicts: 1
    
    
  2. twisted/trial/test/ordertests.py
    1. Does the emacs header make sense in this file? Will it cause any problem for people using emacs test discovery? +# -*- test-case-name: twisted.trial.test.test_script -*-
    2. Actually I see that other fake test modules have that header too so ignore that comment eg twisted/trial/test/erroneous.py
    3. I read your discussion with tom. Here are some thoughts:
      1. I understand that you don't want to further complicate this branch and I'd be happy for it to be merged as is. (especially since other trial tests use hard coded fake test modules)
      2. And actually isn't it a useful that the fake tests can be found and run using trial? During the review, I ran trial twisted/trial/test/ordertests.py using trunk trial, to compare the running order.
      3. ticket:6499#comment:18 is a similar conversation and #6578 was raised as a result.
      4. I've got a ScriptLoader class in source:branches/testable-examples-6362-5/twisted/test/testutils.py#L206 for loading example scripts.
      5. If you wrote your fake TestCases using mktemp, that ScriptLoader might help to add it to the path and then remove it after the test.
      6. Perhaps it would be ok if ordertests.py was in a non-package twisted/test/_data subfolder (without init.py). Then add it to the path in your tests.
  3. doc/core/man/trial.1
    1. minor: You updated the copyright, but the man page still says "Oct 2007" at the bottom.

Its a +1 from me if you merge forward, resolve the conflicts and get
some clean build results...and discuss with Tom whether its ok to
tackle the "test-data in the source." issue in another ticket.

-RichardW.

comment:38 Changed 13 months ago by julian

  • Author changed from tomprince to julian, tomprince
  • Branch changed from branches/trial-order-5787 to branches/trial-order-5787-2

(In [38962]) Branching to trial-order-5787-2.

comment:39 Changed 13 months ago by Julian

Thanks for the thorough review as usual Richard :).

Merged forward, forced at a build. Looks like no failures other than expected twistedchecker complaints.

You make a good point about trial test data for these kinds of tests, although I have to think about whether I agree it outweighs the other general benefit of #6535, which I think is locality to the test that is using the data (and proximity from other stuff that wants to repurpose existing data in potentially fragile ways).

I'll double check with Tom and then merge if he's OK with it after thinking about what a good ticket might be here for the test data cleanup.

comment:40 Changed 13 months ago by exarkun

There's already some line-finding code in trial - the implementation of SynchronousTestCase.flushWarnings includes it. It would be good to refactor and avoid some duplication between that code and the new _maybeFindSourceLine function being added to twisted/scripts/trial.py in this branch. Not necessarily before resolving this ticket, but hopefully soon afterwards. This is some pretty hairy code that needs to account for some gross corner-cases (and I notice the two methods do so slightly different).

comment:41 Changed 13 months ago by exarkun

Oh, I also noticed that test_help_ordersPrintsSynopsisAndQuits interpolates some arbitrary strings into a regular expression and then expects it to make sense. You need to escape those strings if you want this to be reliable. Also, the loop gives no indication of which item is bogus when it fails, so this test will be really obnoxious to debug when it fails.

comment:42 Changed 13 months ago by Julian

Thanks for finding those two. Fixed. One final build which passes besides the usual temperamental buildbots. Merge incoming.

To summarize additional work after this ticket:

  • Add ITestSorter (#6450)
  • (Re)implement --random using --order (#6451)
  • Refactor _maybeFindSourceLine and flushWarnings to use a common line-finding function (#6603)

comment:43 Changed 13 months ago by julian

  • Resolution set to fixed
  • Status changed from new to closed

(In [38971]) Merge trial-order-5787-2: add --order to trial

Author: Julian
Reviewers: cyli, exarkun, tom.prince, thijs, rwall
Fixes: #5787

Allow specification of test execution order via a command line argument.
Currently alphabetical (the default) and by order defined in the source files
are supported.

Note: See TracTickets for help on using tickets.