RFC: Test Collections—Declarative approach to subtests

[dsherret]

#10771 proposed the idea of adding subtests, hooks, and concurrent test execution via a mixed declarative/imperative API similar to what is found in the Go programming language. This proposal is going to explore a similar solution, but keeping the test definitions declarative expanding on @caspervonb's comment. Tests would continue to work in three separate phases of defining, filtering, and finally executing tests.

Interface

The TestDefinition interface is the same, but a new TestCollection interface is added that may be provided to Deno.test.

declare namespace Deno {
  export interface TestDefinition {
    name: string;
    fn: () => void | Promise<void>;
    /** Does not run this test when `true`. */
    ignore?: boolean;
    /** If at least one test has `only` set to true, only run tests that have
     * `only` set to true and fail the test suite. */
    only?: boolean;
    /** Check that the number of async completed ops after the test is the same
     * as number of dispatched ops. Defaults to true.*/
    sanitizeOps?: boolean;
    /** Ensure the test case does not "leak" resources - ie. the resource table
     * after the test has exactly the same contents as before the test. Defaults
     * to true. */
    sanitizeResources?: boolean;
    /** Ensure the test case does not prematurely cause the process to exit,
     * for example via a call to `Deno.exit`. Defaults to true. */
    sanitizeExit?: boolean;
  }

  export interface TestCollection {
    name: string;
    cases: Array<TestDefinition | TestCollection>;
    /** Hook to run before the descendant test definitions are executed. */
    before?: () => void | Promise<void>;
    /** Hook to run after the descendant test definitions are executed.
     * This is executed regardless of whether the descendant tests pass. */
    after?: () => void | Promise<void>;
    /** Does not run this test collection when `true`. */
    ignore?: boolean;
    /** If set to true, only runs tests in this test collection unless `only` is `true`
     * on at least one test definition in the test suite. */
    only?: boolean;
    /** Executes asynchronous sub tests at the same time when able. This will automatically move
     * sanitization checks to a higher level based on the descendant test definitions.
     * For more accurate reporting of which sub test is leaky, set this to `false`. Setting this to
     * `true` may affect sub test execution order. Defaults to `false`. */
    concurrent?: boolean;
  }

  export function test(test: TestDefinition | TestCollection);
  export function test(name: string, fn: () => void | Promise<void>): void;
}

Behaviours

This document will show many of the same examples as #10771 and largely mirror (copy/paste) the text from there, but written with this proposed declarative API.

Sub tests can be defined by providing a test collection to Deno.test:

/** Test with subtests (sequential), but no initalizers or destructors. */
const cases = [
  [1, 1, 2],
  [1, 2, 3],
  [-1, 2, 1],
];
Deno.test({
  name: "addition (sequential)",
  cases: cases.map(([a, b, expected]) => ({
    name: `${a}+${b}=${expected}`,
    fn: () => {
      const actual = a + b;
      assertEquals(actual, expected);
    },
  })),
});

Sub tests can also run concurrently and the resource sanitizers do not need to be disabled as the test executer can move that responsibility to a higher level based on the descendant test definitions (more on that below):

/** Test with subtests (concurrent), but no initializers or destructors. */
Deno.test({
  name: "addition (concurrent)",
  cases: cases.map(([a, b, expected]) => ({
    name: `${a}+${b}=${expected}`,
    fn: async () => {
      const actual = a + b;
      await delay(100);
      assertEquals(actual, expected);
    },
  })),
  concurrent: true, // note this
});

Test collections can have descendant test collections:

Deno.test({
  name: "group with subgroup",
  cases: [{
    name: "subgroup",
    cases: [{
      name: "case 1",
      fn: () => {},
    }, {
      name: "case 2",
      fn: () => {},
    }]
  }],
});

Setup and teardown steps are not as convenient as an imperative approach out of the box...

let database: Database;

Deno.test({
  name: "database tests",
  before: async () => database = await setupDatabase(),
  after: () => database.destroy(),
  cases: [
    {
      name: "first case using database",
      fn: async () => {
        assertEquals(await database.getData(), "data");
      }
    },
    { name: "second case using database", fn: () => {} },
  ],
});

...however, this inconvenience can be mitigated in many ways by userland testing libraries (who would be the consumers of this API).

Only running a specific sub test is possible by adding only: true to the test definition similar to today, but this could also be added to a test collection. Similarly, entire test collections can be ignored by adding ignored: true. See the interface JS docs above for more details.

Sanitizers

Sanitizers (sanitizeOps, sanitizeResources, sanitizeExit) would remain at the test definition level, but if a user specifies concurrent: true on a test collection, then the test execution phase would analyze the descendant test definitions to figure out which tests it could run concurrently based on the test definitions. It would then run all these tests in groups and do sanitization checks after each group ran. In most cases, the entire test collection would be run concurrently and the sanitization check would occur afterwards.

This has the obvious downside of reduced resolution on which test is leaking, however an instructive message can be given to the user on a failure to tell them to run the tests again with concurrent: false to drill down to the specific test or the tests could be run again serially when this occurs in order to tell them which test it is (doesn't always work if it's flaky). Either way, this may provide a good user experience because it does not require manually disabling a lot of test case sanitizers.

Permissions

These would work the same way as sanitizers and the test executer would be able to figure out which tests can run concurrently based on the defined test definitions that it knows about.

Reporter Output

Largely the same as #10771, except accurate numbers on how many tests are left to execute and how many total tests there are before filtering are possible to report to the user.

Filtering

Filtering would work well and fast because filtering happens after defining tests and before executing them or their setup code. "Only" is also possible because of this. Additionally filtering based on a substring of a subtest name is possible.

Shuffling

Shuffling sub tests would be possible with this approach.

A Building Block

Similarly to #10771, this API is directed towards testing framework authors (ex. using describe and it like mocha). It would not be very convenient for end users to use this API.

cc @lucacasonato @caspervonb

[bartlomieju]

In my view this proposal is the preferred way forward instead of #10771. It doesn't introduced "mixed" way to define tests (we stick to the declarative one), while vastly expanding ability for 3rd party testing frameworks to integrate into Deno.

I would maybe bike-shed on the TestDefinition.entries name - I think cases would be better.

[dsherret]

I like cases better too. Updated.

[caspervonb]

Ooo, bikeshedding...! The name of the type is definition... definitions? 😬

[dsherret]

I don't have a strong opinion on the name, but it possibly also contains collections. I do like cases a bit more because it's shorter.

[caspervonb]

Am for declarative over procedural altho the before and after hooks here are kinda moot with seemingly little wins over having the user wrap their functions?

Strawman and just winging this but maybe we could provide some sort of wrapping executor instead? This at-least gives you a closure scope and extension point for prettier things in std/testing (for example, pass along some user defined context).

type TestExecutor = (test: TestDefinition) => void | Promise<void>;

export interface TestCollection {
  definitions: TestDefinitions;
  executor?: TestExecutor;
}

Deno.test({
  definitions: [
    // ...
  ],
  async executor(test: TestFunction) {
    const someState = ...;
    await definition.fn(someState);
    someState.close();
  },
});

[dsherret]

We want some way to execute user defined setup and teardown code without affecting the santizer checks. Additionally, we also want a way to only execute some user code after all the tests are run (not after each test has run). The user wrapping their functions doesn't help with either of those, but before/after hooks do.

[caspervonb]

Just callbacks like this encourages global scope tho, which isn't awesome and offers little over just preparing the state ahead of time, resetting it before each run which would keep sanitizers in check too.

The user wrapping their functions doesn't help with either of those, but before/after hooks do.

It does tho, as you get to execute something before and after the sanitisers.

[caspervonb]

Just to clarify a bit here, sanitisers and such are baked into the test function by the test harness during the registration phase.

So if we have a optional function hook on a test group that the runner uses to invoke each test rather than calling it directly, we can then change up the parameters that is passed to each test function which is a much better building block imo than just having some functions that can only register/unregister things in the global scope.

That would be our before and after "each" hook in one single optional function, we can also consider having another executor to represent before and after all.

Plenty of options here to explore, but I'd prefer if we see how far we get without hooks as the first pass.

[dsherret]

I meant if the user just wraps their functions then they can't execute something before and after the santizers. If we provide an executor like you are mentioning I can now see that would work, however I'm not sure how this would be easily typed with TypeScript. I tried it out a little bit and in the end it seems like before and after hooks would just be more clear/less complicated and is already very common in JS testing frameworks. Additionally with an executor what happens if descendants beyond the children test definitions want access to the created resources?

Do you think you could show an example of how the executor implementation might look in the TS playground with all the types set (ex. current proposal state)?

[zandaqo]

Speaking of hooks, what about beforeEach/afterEach hooks? Are they left to third-party frameworks?

I assume one could mimic beforeEach/afterEach by making each case a subgroup, though, that's a tad bit hackish:

let someObject = undefined;
const beforeEach = () => { someObject = {}; };
const afterEach = () => { someObject = undefined; };

Deno.test({
  name: "group with subgroup",
  cases: [{
    name: "subgroup",
    before: beforeEach;
    after: afterEach;
    cases: [{
      name: "case 1",
      fn: () => {},
    },
{
    name: "subgroup",
    before: beforeEach;
    after: afterEach;
    cases: [{
      name: "case 1",
      fn: () => {},
    }]
  }],
});

[dsherret]

Yes, they'd be left to the user and third party frameworks.

You wouldn't need to create a subgroup and instead the function that's executed could be wrapped. See Casper's comment here for an example: #10486 (comment)

[zandaqo]

@dsherret In other words, without a third party framework, we would declaratively define our beforeAll/afterAll hooks for groups, but imperatively wrap our test functions with beforeEach/afterEach hooks. While I appreciate the simplicity of it, I think it would be confusing for users coming from other test runners where both types of hooks are defined similarly.

[dsherret]

This proposal is more meant for a third party framework to implement stuff like nested describes, beforeAll, etc all with filtering and exection by deno test. They'll be able to implement beforeEach/afterEach without explicit beforeEach/afterEach hooks in the TestCollection. Regardless, having beforeEach/afterEach hooks explicitly are something that could be added easily in the future.

[caspervonb]

Just want to point out, that declarative maintains our clean separation between the test harness and test runner.

While it's not something we're committed to maintain by any means, this clean separation let me build this little experimental tool to run tests in browsers simply by proxying the module registration with a stub test harness and re-use the test runner in the binary build of demo completely because the two are completely separated.