build: test parallelization ⚡️

[erickzhao]

Summary

This PR streamlines our CircleCI configuration in two ways.

Build once, test everywhere

In main, our CircleCI job matrix runs two test jobs for each target platform and architecture. In practice, this means 8 test jobs:

• fast-tests-arm64-node/macos
• fast-tests-x64-node/macos
• fast-tests-x64-node/windows
• fast-tests-x64-node/linux
• slow-tests-arm64-node/macos
• slow-tests-x64-node/windows
• slow-tests-x64-node/macos
• slow-tests-x64-node/linux

Each one of these jobs currently the entire linting and building process for Forge separately. These are both wasteful:

• Lint: Static analysis of the code only needs to be performed once.
• Build: Being pure JS, Forge only needs to be compiled a single time.

Thus, this PR adds a separate lint-and-build job that runs on Linux because the runner on that platform is fast and cheap.

This will reduce total CI time from $t_{total} = t_{b1} + t_{t1} + ... + t_{b8} + t_{t_8}$ to $t_{total}' = t_{b} + t_{t1} + ... + t_{t8}$, where $t_{bn} + t_{tn}$ is the build and test time respectively for job ${n}$.

However, in more practical terms, this will only reduce time-to-green by $\Delta = t_{slowest\ build} - t_{fastest\ build}$.

CircleCI test splitting

CircleCI has a test splitting feature that relies on submitting JUnit format test reports to Circle. #3316 added that reporting, and this PR follows it up by adding the test splitting feature to our Slow test suites.

In an ideal world where tests can be split evenly $n$ ways, this theoretically reduces the runtime of slow test jobs from $t_{slow}$ to $t_{slow}'$, where $n$ is the parallelization factor for the runners.

$t_{slow} = t_{overhead} + t_{tests}$
$t_{slow}' = t_{overhead} + \dfrac{t_{tests}}{n}$

In order to do so, a few things needed to be changed:

• test-globber.ts was removed for some tweaks in .mocharc.js. yarn test:fast and yarn test:slow should still work as before, though.
• Windows slow test CI code was simplified and streamlined with the rest of the Linux and macOS setup.

Outcomes

In practice, the first green run in this PR reduced the time-to-green time from 32:42 -> 25:58 (-20.6%). There are a few future improvements to explore:

• Windows slow tests are really slow due to setup time. Half of the 23-minute runtime is dedicated to installing node modules and setting up wixtoolset via Chocolatey.
• Something's wrong with the JUnit format and test splitting doesn't seem to work via timing data. More investigation is required to figure out how to get this working. This means that the parallelization of slow tests is not optimized.
• I only applied $n=2$ test runners in parallel in this iteration of the PR. The number of runners is a tradeoff between time-to-green and total CI runtime.

[VerteDinde]

Amazing - thanks so much for adding this! 👏

[erikian]

🚀