Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: adding some basic documentation #10

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 126 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,129 @@ A tool for managing many combinations of real and fake NPM packages within your

Extracted from [ember-auto-import](https://github.com/ef4/ember-auto-import). No docs here yet, but:
- you can see how it's used in ember-auto-import/test-scenarios
- it's all typescript so you can follow the types to learn what's possible
- it's all typescript so you can follow the types to learn what's possible

## Getting started

`scenario-tester` works especially well with a monorepo. We recommend that you create a new workspace package for your scenarios e.g. :

```
mkdir test-packages/test-scenarios
```

Next you need to create some scenarios. The best way to create a scenario is from a Project, and the best way to create a project is to have a base project on disk. For example, in EmberJS you can create a new project with `ember new base-project` and it will create a new empty project for you. We can use this as a starting point for our other scenarios.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably want --skip-npm and --skip-git flags on that ember command

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did that when I personally ran the command but I didn't want to make this all about ember 🤔 there is nothing tying scenario-tester to ember and I thought that was an extra detail that we shouldn't include here. Does that make sense?

I don't feel all that strongly about this so if you (or anyone) does I'm happy to add them 👍

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does make sense, but as is, the example instruction of ember new base-project is misleading / wrong, which is where I'm coming from.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the purpose of having a less frustrating experience and a working set of commands I would add them
OR provide a command to achieve the same without using the ember-cli


```
cd test-packages/test-scenarios
ember new base-project
```

Next you can create a scenarios file that you can export a base project function:


```js
// test-packages/tests-scenarios/scenarios.js
import { Project } from 'scenario-tester';
import { dirname } from 'path';

// This is a way to allow for `require.resolve` to work in esm modules in node.
// If you are using CJS you can skip these two lines
import { createRequire } from 'node:module';
const require = createRequire(import.meta.url);

export function baseApp() {
return Project.fromDir(dirname(require.resolve('./classic-app-template/package.json')), { linkDevDeps: true });
}
```

While it's not quite that useful (yet) you can use this base app to build a specific test scenario. Let's do a simple test that boots your app.

```js
// test-packages/test-scenarios/boot-app-test.js

import { baseApp } from './scenarios';
import { Scenarios } from 'scenario-tester';
import qunit from 'qunit';

const { module: Qmodule, test } = qunit;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we alias these?
I don't think we need to in ESM?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so in a real ESM setup you absolutely do because qunit isn't ESM 😞 In typescript we have been getting away with it for a while because it's so loosey-goosey with the whole thing 🤪

I've been testing this in #11 and ember-fastboot/ember-cli-fastboot#919 and it doesn't work any other way 👍

Copy link

@NullVoxPopuli NullVoxPopuli May 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so in a real ESM setup you absolutely do because qunit isn't ESM

can you expand on this? I've been using module with qunit in a real ESM environment (vite) no problem.

I maybe thought that this was a browser vs node issue, but even in node, seems like it should be fine:
image

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NullVoxPopuli vite papers over a lot of incompatibilities for you so not sure if it's a good base to go off of

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh sorry I thought you were talking about something else 🤔 I thought you were just telling me to do this:

import { module as Qmodule, test } from 'qunit';

and that's what I was saying just doesn't work.

I didn't realise you were commenting on the fact that I was renaming module 🙃 The reason I did this is because I copy and pasted from... either ember-auto-import or embroider... I can't remember.

I never questioned it, but if it's not an issue I am happy to just use module directly 🤷 I assume Ed probably used it because there were a billion and a half uses of the word module in the codebase and it is easier to follow 😂


Scenarios.fromProject(baseApp)
.map('boot-app-test', project => {
// TODO add some specifics for your test scenarios here
})
.forEachScenario(scenario => {
Qmodule(scenario.name, function (hooks) {
let app; // PreparedApp

hooks.before(async () => {
app = await scenario.prepare();
// any custom setup that you have for each scenario
});

test('it works', async function (assert) {
// your custom test code
assert.ok(true);
});
});
});
```

The power in this kind of setup comes when you start adding in different dependencies that you might want to test this app with. We'll go into that a bit more in the next section. For now to run these tests we can run the following command.

```bash
npx qunit *-test.js

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would favour package.json scripts even though npx may function
as I've seen funny things with npx in a pnpm project where npx ember failed to run spectacularly but npm script of the exact same was just fine

```

This will run each of the tests in your setup sequentially. This can take a while, so on CI it can be useful to run all your scenarios in paralell. For this we need your CI to be able to discover all the possible tests that are available and run one specific test.

**Note:** all of our examples are using Gihub Actions so your CI provider may need a different setup

## CI Setup

The first job in your CI will need to discover all the tests that are available to use:

```yaml
jobs:
discover_matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '16'
cache: npm
- run: npm install
- id: set-matrix
working-directory: test-packages/test-scenarios
run: echo "::set-output name=matrix::$(npm run --silent test:list -- --matrix 'npm run test -- --filter %s:')"
```

This will generate a matrix for you that will contain a command to filter down to just one test. You can then use this to run the test in another job:

```yaml
jobs:
scenarios:
needs: discover_matrix
name: ${{ matrix.name }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix: ${{fromJson(needs.discover_matrix.outputs.matrix)}}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '16'
cache: npm
- run: npm ci
- name: test
run: ${{ matrix.command }}
working-directory: test-packages/test-scenarios
```