chore: Run tests in parallel jobs

[KapJI]

Changes

• Add -filter flag to TestScript which allows to filter scripts by filename using regex. This can be useful for tests sharding and running individual test locally.
• go test -short runs all tests except TestScript.
• Split tests into 2 jobs:
  • test-index: 0 runs ./... -short and TestScript tests ^[a-h]
  • test-index: 1 run TestScript remaining tests ^[i-z]
• Split macos tests into 3 jobs:
  • test-index: 0 runs ./... -short and TestScript tests ^[a-d]
  • test-index: 1 run TestScript remaining tests ^[e-i]
  • test-index: 2 run TestScript remaining tests ^[j-z]

Results

• Split all tests into 2 shards: Workflow completion is down from 9m30s to 6m20s, 33% faster.
• Split macos tests into 3 shards: Workflow completion is down from 9m30s to 4m58s, 47% faster.

And this is where we started: 22m55s, 78% improvement!

[twpayne]

Hey, I really appreciate your work here! At the same time, is the runtime of the tests dominated by running the tests themselves or is the runtime dominated by all the cache loading/compiling that has to happen before? If the test time dominates then splitting the tests makes sense. If not...

[KapJI]

It seems that listing (or rather building test binaries) takes around 1 minute. Remaining 5-6 minutes is test execution.

My goal here is to reduce test jobs run time to about 3 minutes, then they will be on par with other CI jobs. This should save another 3 minutes for workflow completion on PRs and releases.

[twpayne]

Thank you! The testscript tests are very I/O heavy. If there's a way to run them on a ramdisk, it might be a significant win.

[KapJI]

If there's a way to run them on a ramdisk, it might be a significant win.

I tried in 7946fad but there is absolutely no difference unfortunately. Maybe it's already a ramdisk.

[halostatue]

This is very clever. The only question I have is whether the test scripts are "balanced" in the way you segmented them (0-9a-h and i-z). I suspect that it doesn't matter much because there's not that many, and it wouldn't be too hard to make more slices if the number of test scripts increases.

[KapJI]

The only question I have is whether the test scripts are "balanced"

The runtime of jobs is pretty close as you can see. Mac jobs are the slowest now, it's pretty easy to split them into 3 parts if needed.

[KapJI]

So this should save additional 4 minutes on every PR and release CI, sounds pretty good.

[twpayne]

Brilliant, thank you!