test_runner: support test level randomization

Author: pmarchini

doc/api/cli.md

@@ -2768,9 +2768,9 @@ option set. This flag is not necessary when test isolation is disabled.
 added: REPLACEME
 -->
 
-Set the seed used to randomize the order in which test files are executed.
-Providing this flag enables randomization implicitly, even without
-`--test-randomize`.
+Set the seed used to randomize test execution order. This applies to both test
+file execution order and queued tests within each file. Providing this flag
+enables randomization implicitly, even without `--test-randomize`.
 
 The value must be an integer between `0` and `4294967295`.
 
@@ -2782,13 +2782,26 @@ This flag cannot be used with `--watch`.
 added: REPLACEME
 -->
 
-Randomize the order in which test files are executed. This can help detect
-tests that rely on shared state or execution order.
+Randomize test execution order. This applies to both test file execution order
+and queued tests within each file. This can help detect tests that rely on
+shared state or execution order.
 
 The seed used for randomization is printed in the test summary and can be
 reused with `--test-random-seed`.
 
-This flag cannot be used with `--watch`.
+For detailed behavior and examples, see the
+[test runner][] documentation, especially the
+`Randomizing test file execution order` section.
+
+#### Limitations
+
+* Randomization is applied when there are multiple queued sibling tests.
+  Patterns that explicitly await each `test()` call (for example,
+  `await test(...)` inside a loop) run subtests sequentially and preserve
+  declaration order, so their execution order is not randomized.
+* Suite-style APIs such as `describe()`/`it()` or `suite()`/`test()` do not
+  disable randomization when sibling tests are queued.
+* This flag cannot be used with `--watch`.
 
 ### `--test-reporter`
 

doc/api/test.md

@@ -593,8 +593,10 @@ added: REPLACEME
 
 > Stability: 1.0 - Early development
 
-The test runner can randomize the order of discovered test files to help detect
-order-dependent tests. Use `--test-randomize` to enable this mode.
+The test runner can randomize execution order to help detect
+order-dependent tests. When enabled, the runner randomizes both discovered
+test files and queued tests within each file. Use `--test-randomize` to
+enable this mode.
 
 ```bash
 node --test --test-randomize
@@ -607,14 +609,68 @@ as a diagnostic message:
 Randomized test order seed: 12345
 ```
 
-Use `--test-random-seed=<number>` to replay the same order in a deterministic
-way. Supplying `--test-random-seed` also enables randomization, so
-`--test-randomize` is optional when a seed is provided:
+Use `--test-random-seed=<number>` to replay the same randomized order
+deterministically. Supplying `--test-random-seed` also enables randomization,
+so `--test-randomize` is optional when a seed is provided:
 
 ```bash
 node --test --test-randomize --test-random-seed=12345
 ```
 
+In most test files, randomization works automatically. One important exception
+is when subtests are awaited one by one. In that pattern, each subtest starts
+only after the previous one finishes, so the runner keeps declaration order
+instead of randomizing it.
+
+Example: this runs sequentially and is **not** randomized.
+
+```mjs
+import test from 'node:test';
+
+test('math', async (t) => {
+  for (const name of ['adds', 'subtracts', 'multiplies']) {
+    // Sequentially awaiting each subtest preserves declaration order.
+    await t.test(name, async () => {});
+  }
+});
+```
+
+```cjs
+const test = require('node:test');
+
+test('math', async (t) => {
+  for (const name of ['adds', 'subtracts', 'multiplies']) {
+    // Sequentially awaiting each subtest preserves declaration order.
+    await t.test(name, async () => {});
+  }
+});
+```
+
+Using suite-style APIs such as `describe()`/`it()` or `suite()`/`test()`
+still allows randomization, because sibling tests are queued together.
+
+Example: this remains eligible for randomization.
+
+```mjs
+import { describe, it } from 'node:test';
+
+describe('math', () => {
+  it('adds', () => {});
+  it('subtracts', () => {});
+  it('multiplies', () => {});
+});
+```
+
+```cjs
+const { describe, it } = require('node:test');
+
+describe('math', () => {
+  it('adds', () => {});
+  it('subtracts', () => {});
+  it('multiplies', () => {});
+});
+```
+
 `--test-randomize` and `--test-random-seed` are not supported with `--watch` mode.
 
 Matching files are executed as test files.
@@ -657,8 +713,10 @@ test runner functionality:
 * `--test-reporter` - Reporting is managed by the parent process
 * `--test-reporter-destination` - Output destinations are controlled by the parent
 * `--experimental-config-file` - Config file paths are managed by the parent
-* `--test-randomize` - File randomization is managed by the parent process
-* `--test-random-seed` - File randomization seed is managed by the parent process
+* `--test-randomize` - Randomization is managed by the parent process and
+  propagated to child processes
+* `--test-random-seed` - Randomization seed is managed by the parent process and
+  propagated to child processes
 
 All other Node.js options from command line arguments, environment variables,
 and configuration files are inherited by the child processes.
@@ -1565,12 +1623,12 @@ changes:
       that specifies the index of the shard to run. This option is _required_.
     * `total` {number} is a positive integer that specifies the total number
       of shards to split the test files to. This option is _required_.
-  * `randomize` {boolean} Randomize the execution order of test files.
+  * `randomize` {boolean} Randomize execution order for test files and queued tests.
     This option is not supported with `watch: true`.
     **Default:** `false`.
-  * `randomSeed` {number} Seed used when randomizing test file order. If this
-    option is set, runs can replay the same randomized file order
-    deterministically, and setting this option also enables randomization.
+  * `randomSeed` {number} Seed used when randomizing execution order. If this
+    option is set, runs can replay the same randomized order deterministically,
+    and setting this option also enables randomization.
     **Default:** `undefined`.
   * `rerunFailuresFilePath` {string} A file path where the test runner will
     store the state of the tests to allow rerunning only the failed tests on a next run.

lib/internal/test_runner/runner.js

@@ -155,6 +155,8 @@ function getRunArgs(path, { forceExit,
                             argv: suppliedArgs,
                             execArgv,
                             rerunFailuresFilePath,
+                            randomize,
+                            randomSeed,
                             root: { timeout },
                             cwd }) {
   const processNodeOptions = getOptionsAsFlagsFromBinding();
@@ -196,6 +198,12 @@ function getRunArgs(path, { forceExit,
   if (rerunFailuresFilePath) {
     ArrayPrototypePush(runArgs, `--test-rerun-failures=${rerunFailuresFilePath}`);
   }
+  if (randomize === true) {
+    ArrayPrototypePush(runArgs, '--test-randomize');
+  }
+  if (randomSeed != null) {
+    ArrayPrototypePush(runArgs, `--test-random-seed=${randomSeed}`);
+  }
 
   ArrayPrototypePushApply(runArgs, execArgv);
 
@@ -682,6 +690,25 @@ function run(options = kEmptyObject) {
       );
     }
   }
+  if (rerunFailuresFilePath) {
+    validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
+    // TODO(pmarchini): Support rerun-failures with randomization by
+    // persisting the randomization seed in the rerun state file.
+    if (randomSeed != null) {
+      throw new ERR_INVALID_ARG_VALUE(
+        'options.randomSeed',
+        randomSeed,
+        'is not supported with rerun failures mode',
+      );
+    }
+    if (randomize) {
+      throw new ERR_INVALID_ARG_VALUE(
+        'options.randomize',
+        randomize,
+        'is not supported with rerun failures mode',
+      );
+    }
+  }
   if (randomize) {
     randomSeed ??= createRandomSeed();
   }
@@ -694,10 +721,6 @@ function run(options = kEmptyObject) {
     );
   }
 
-  if (rerunFailuresFilePath) {
-    validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
-  }
-
   if (shard != null) {
     validateObject(shard, 'options.shard');
     // Avoid re-evaluating the shard object in case it's a getter
@@ -794,6 +817,8 @@ function run(options = kEmptyObject) {
     functionCoverage: functionCoverage,
     cwd,
     globalSetupPath,
+    randomize,
+    randomSeed,
   };
 
   const root = createTestTree(rootTestOptions, globalOptions);

lib/internal/test_runner/test.js

@@ -10,6 +10,7 @@ const {
   ArrayPrototypeUnshiftApply,
   Error,
   FunctionPrototype,
+  MathFloor,
   MathMax,
   Number,
   NumberPrototypeToFixed,
@@ -47,6 +48,7 @@ const { MockTracker } = require('internal/test_runner/mock/mock');
 const { TestsStream } = require('internal/test_runner/tests_stream');
 const {
   createDeferredCallback,
+  createSeededGenerator,
   countCompletedTest,
   isTestFailureError,
   reporterScope,
@@ -648,11 +650,16 @@ class Test extends AsyncResource {
     this.message = typeof skip === 'string' ? skip :
       typeof todo === 'string' ? todo : null;
     this.activeSubtests = 0;
+    this.subtestQueueRandom = this.config.randomize ?
+      createSeededGenerator(this.config.randomSeed) :
+      null;
     this.pendingSubtests = [];
     this.readySubtests = new SafeMap();
     this.unfinishedSubtests = new SafeSet();
     this.subtestsPromise = null;
     this.subtests = [];
+    this.nextReportOrder = 1;
+    this.reportOrder = 0;
     this.waitingOn = 0;
     this.finished = false;
     this.hooks = {
@@ -776,13 +783,37 @@ class Test extends AsyncResource {
     ArrayPrototypePush(this.pendingSubtests, deferred);
   }
 
+  /**
+   * Ensure each subtest has a contiguous, per-parent reporting order.
+   * This is assigned at dequeue time for randomized runs, but tests that are
+   * cancelled before dequeue still need an order to be reported.
+   * @param {Test} subtest
+   * @returns {void}
+   */
+  assignReportOrder(subtest) {
+    if (subtest.reportOrder === 0) {
+      subtest.reportOrder = this.nextReportOrder++;
+    }
+  }
+
+  dequeuePendingSubtest() {
+    if (!this.subtestQueueRandom || this.pendingSubtests.length < 2) {
+      return ArrayPrototypeShift(this.pendingSubtests);
+    }
+
+    // Pick a uniformly random pending sibling when randomization is enabled.
+    const index = MathFloor(this.subtestQueueRandom() * this.pendingSubtests.length);
+    return ArrayPrototypeSplice(this.pendingSubtests, index, 1)[0];
+  }
+
   /**
    * @returns {Promise<void>}
    */
   async processPendingSubtests() {
     while (this.pendingSubtests.length > 0 && this.hasConcurrency()) {
-      const deferred = ArrayPrototypeShift(this.pendingSubtests);
+      const deferred = this.dequeuePendingSubtest();
       const test = deferred.test;
+      this.assignReportOrder(test);
       test.reporter.dequeue(test.nesting, test.loc, test.name, this.reportedType);
       await test.run();
       deferred.resolve();
@@ -794,7 +825,8 @@ class Test extends AsyncResource {
    * @returns {void}
    */
   addReadySubtest(subtest) {
-    this.readySubtests.set(subtest.childNumber, subtest);
+    this.assignReportOrder(subtest);
+    this.readySubtests.set(subtest.reportOrder, subtest);
 
     if (this.unfinishedSubtests.delete(subtest) &&
         this.unfinishedSubtests.size === 0) {
@@ -1011,6 +1043,7 @@ class Test extends AsyncResource {
       return deferred.promise;
     }
 
+    this.parent.assignReportOrder(this);
     this.reporter.dequeue(this.nesting, this.loc, this.name, this.reportedType);
     return this.run();
   }
@@ -1315,7 +1348,7 @@ class Test extends AsyncResource {
   isClearToSend() {
     return this.parent === null ||
       (
-        this.parent.waitingOn === this.childNumber && this.parent.isClearToSend()
+        this.parent.waitingOn === this.reportOrder && this.parent.isClearToSend()
       );
   }
 

src/node_options.cc

@@ -988,15 +988,15 @@ EnvironmentOptionsParser::EnvironmentOptionsParser() {
             kAllowedInEnvvar,
             OptionNamespaces::kTestRunnerNamespace);
   AddOption("--test-randomize",
-            "run test files in a random order",
+            "run tests in a random order",
             &EnvironmentOptions::test_randomize,
             kAllowedInEnvvar,
             false,
             OptionNamespaces::kTestRunnerNamespace);
   AddOption(
       "[has_test_random_seed]", "", &EnvironmentOptions::has_test_random_seed);
   AddOption("--test-random-seed",
-            "seed used to randomize test file execution order",
+            "seed used to randomize test execution order",
             &EnvironmentOptions::test_random_seed,
             kAllowedInEnvvar,
             OptionNamespaces::kTestRunnerNamespace);

test/fixtures/test-runner/output/randomize_nested_scenarios_output_cli_none.js

@@ -0,0 +1,17 @@
+'use strict';
+require('../../../common');
+const fixtures = require('../../../common/fixtures');
+const spawn = require('node:child_process').spawn;
+
+spawn(
+  process.execPath,
+  [
+    '--no-warnings',
+    '--test-reporter', 'spec',
+    '--test-random-seed=1',
+    '--test-isolation=none',
+    '--test',
+    fixtures.path('test-runner/randomize/internal-order-nested-scenarios.cjs'),
+  ],
+  { stdio: 'inherit' },
+);