Opened 10 years ago

Closed 4 years ago

#1948 defect closed duplicate (duplicate)

Trial's TestLoader should sort tests in a specfic order

Reported by: jml Owned by:
Priority: normal Milestone:
Component: trial Keywords:
Cc: Branch:


It should clearly and cleanly provide:

  • Sorting alphabetically
  • Sorting reverse alphabetically
  • Sorting randomly

The sorting code should be basically separate from the loading code.

Attachments (1)

trial-order.diff (1.9 KB) - added by warner 7 years ago.
basic patch to leave test cases in argv order

Download all attachments as: .zip

Change History (8)

comment:1 Changed 10 years ago by therve

#2559 was a duplicate of this.

comment:2 Changed 10 years ago by warner

I'd like to add "Sorting according to argv[]" to the list of options. I have frequently run into tricky failures that are uncovered while --random was enabled. In trying to isolate which pair of tests had the problem, I want to use 'trial TEST1 TEST2 TEST3', and remove tests one at a time. But without a way to preserve the order that I specify via sys.argv, I have no way to run a set of tests in the same order as the --random pass chose.

Note that --random=origseed won't help, because on later runs it is shuffling an entirely different list of test cases. Now, if we used --random to seed a consistent hashing -based permutation, this might Just Work, but I think letting people use sys.argv in the obvious way would be less surprising.

comment:3 Changed 7 years ago by Screwtape

  • Summary changed from TestLoader's sorting is a mess to Trial's TestLoader should sort tests in a specfic order

#3088 (and hence its dupe #3249) were duplicates of this.

Resummarizing to hopefully make this bug easier to find (old title was "TestLoader's sorting is a mess")

Changed 7 years ago by warner

basic patch to leave test cases in argv order

comment:4 Changed 7 years ago by warner

that patch I just attached has the following drawbacks:

  • requires O(n2) time to avoid duplicates (but really, how many test-case arguments do you intend to add?)
  • no tests yet

comment:5 Changed 7 years ago by exarkun

I also frequently want the argv ordering feature.

comment:6 Changed 6 years ago by <automation>

  • Owner jml deleted

comment:7 Changed 4 years ago by exarkun

  • Resolution set to duplicate
  • Status changed from new to closed

This ticket is perhaps now obsolete?

  • #5520 implemented running in command line order.
  • #5787 proposes implementing a --order option supporting arguments such as "toptobottom". "alphabetical" would fit in with this.
Note: See TracTickets for help on using tickets.