The tests/
folder is the place for everything related to testing of quarto-cli
.
We run several type of tests
- Unit tests, located in
unit/
folder - Integration tests, located in
integration/
folder - smoke tests localed in
smoke
folder
Tests are run in our CI workflow on GHA at each commit, and for each PR.
Tests are running through Deno.test()
framework, adapted for our Quarto project and all written in Typescript. Infrastructure are in tests.ts
, tests.deps.ts
verify.ts
and utils.ts
which contains the helper functions that can be used.
unit/
andintegration/
,smoke/
contain some.ts
script representing each tests.docs/
is a special folder containing of the necessary files and projects used for the tests.
Here are what is expected in the environment for the tests :
- R should be installed and in PATH - rig is a good tool to manage R versions. e.g
rig install 4.4.2
andrig default 4.4.2
to install and set the version to 4.4.2- On Windows, Rtools should be too (for source package installation)
- Python should be installed and in PATH - pyenv is a good option to manage Python versions.
- On Windows, it will be
pyenv-win
to manage versions. Otherwise or install from https://www.python.org/ manually or usingwinget
.
- On Windows, it will be
- Julia should be installed and in PATH - juliaup is a good option to manage Julia versions.
- On Windows, one way is using
winget install julia -s msstore
and then add%LOCALAPPDATA%/Programs/Julia/bin
to PATH
- On Windows, one way is using
Running tests require to have a local environment setup with Quarto development, TinyTeX, R, Python and Julia.
To help with this configuration, the tests/
folder contains configure-test-env.sh
and configure-test-env.ps1
. It will check for the tools and update the dependencies to what is used by Quarto tests.
Running the script at least one will insure you are correctly setup. Then, it is run as part of running the tests so that dependencies are always updated. Set QUARTO_TESTS_NO_CONFIG
to skip this step when running tests.
Dependencies are managed using the following tools:
We use renv. renv.lock
and renv/
folders are the files used to recreate the environment for R.
Updating renv.lock
is done using renv::snapshot()
. File shouldn't be modified manually.
Our project is using explicit dependencies discovery through a DESCRIPTION
file. This is to avoid a costly scanning of all files in tests/
to guess R dependencies. This means that if you need to add a test with a new R package dependencies:
- Add package(s) to
DESCRIPTION
intests/
renv::install()
the package into the project library- Finish to work on your test
renv::snapshot()
to record the new dependency in therenv.lock
- Commit the new
DESCRIPTION
andrenv.lock
See documentation if you need to tweak the R environment.
After a dependency update, you can run configure-test-env.sh
or configure-test-env.ps1
to update the environment, or manually run renv::restore()
to recreate the environment with new versions. Be sure to update your R version if needed.
We now use uv (previously, it was pipenv) to manage dependencies and recreate easily on all OS. uv
will not be installed as part of the configuration - so it needs to be installed manually - see various way at: https://docs.astral.sh/uv/getting-started/installation/
uv will handle the python versions, including its installation, based on the .python-version
we have in tests/
folder. It will also manage the virtual environment in .venv
folder.
A virtual environment will be created locally in .venv
folder (ignored on git) and activated when running tests. uv run
can help activating the environment outside of running tests to run a command in the environment.
pyproject.toml
contains our dependencies requirement for the tests project. It can be manually updated but it is best to just use uv
commands. For instance, adding a new dependency can be done with uv add plotly
and it will update the file, update the uv.lock
and install in the virtual environment. uv.lock
should never be updated manually, and it is tracked by git, as it allows to recreate the exact environment on different environment (Linux, Mac, Windows, locally and on CI).
See other uv
command if you need to do more.
For a change of python versionn, .python-version
needs to be updated, and then uv
will take care of the rest. configure-test-env
script will check for uv
and if installed, it will called uv sync
to make sure the project virtual environment is up to date with the lockfile.
Note that ./run-test.ps1
and .run-tests.sh
:
- run
configure-test-env
script by default, unlessQUARTO_TESTS_NO_CONFIG
environment variable is set to a non-empty value. - Activate the local virtualenv espected in
.venv
. SetQUARTO_TESTS_FORCE_NO_VENV
to a non-empty value to prevent this behavior. (It replacesQUARTO_TESTS_FORCE_NO_PIPENV
which still is considered for backward compatibility but deprecated)
Julia uses built-in package manager Pkg.jl- we provide Project.toml
and Manifest.toml
to recreate the environment.
Project.toml
contains our direct dependency and Manifest.toml
is the lock file that will be created (Pkg.resolve()
).
See documentation on how to add, remove, update if you need to tweak the Julia environment.
Tests are run using run-tests.sh
on UNIX, and run-tests.ps1
on Windows.
# run all tests
./run-tests.sh
# run a specific tests file
./run-tests.sh smoke/extensions/extension-render-doc.test.ts
# run all tests
./run-tests.ps1
# run a specific tests file
./run-tests.ps1 smoke/extensions/extension-render-doc.test.ts
Those files will run configure-test-env
scripts to check for requirements and set up dependencies (except on Github Action as this is done in the workflow file directly).
You can prevent test configuration locally by setting QUARTO_TESTS_NO_CONFIG
environment variable to a non-empty value.
QUARTO_TESTS_NO_CONFIG="true" ./run-tests.sh
$env:QUARTO_TESTS_NO_CONFIG=$true
./run-tests.ps1
docs/smoke-all/
is a specific folder to run some tests written directly within .qmd
, .md
or .ipynb
files (but files starting with _
will be ignored). They are run through the smoke/smoke-all.tests.ts
script. To ease running smoke-all tests, run-tests.sh
has a special behavior where it will run ./smoke/smoke-all.tests.ts
when passed a .qmd
, .md
or .ipynb
file, not starting with _
.
# run tests for all documents in docs/smoke-all/
./run-tests.sh smoke/smoke-all.tests.ts
# run tests for some `.qmd` document in a specific place (using glob)
./run-tests.sh docs/smoke-all/2022/**/*.qmd
# or using longer version
./run-tests.sh smoke/smoke-all.test.ts -- docs/smoke-all/2022/**/*.qmd
# run test for a specific document
./run-tests.sh docs/smoke-all/2023/01/04/issue-3847.qmd
# or using using longer version
./run-tests.sh smoke/smoke-all.test.ts -- docs/smoke-all/2023/01/04/issue-3847.qmd
Examples of tests output after it ran
$ ./run-tests.sh smoke/smoke-all.test.ts -- docs/smoke-all/2023/01/04/issue-3847.qmd
> Checking and configuring environment for tests
>>>> Configuring R environment
* The library is already synchronized with the lockfile.
>>>> Configuring Python environment
Setting up python environnement with pipenv
Installing dependencies from Pipfile.lock (0ded54)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
>>>> Configuring Julia environment
Setting up Julia environment
Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/e32a90da027ca45d84678b826fffd3110bb3fc90/build.log`
Building IJulia → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/59e19713542dd9dd02f31d59edbada69530d6a14/build.log`
>>>> Configuring TinyTeX environment
Setting GH_TOKEN env var for Github Download.
tinytex is already installed and up to date.
> Activating virtualenv for Python tests
Check file:///home/cderv/project/quarto-cli/tests/smoke/smoke-all.test.ts
running 1 test from ./smoke/smoke-all.test.ts
[smoke] > quarto render docs/smoke-all/2023/01/04/issue-3847.qmd --to html ...
------- output -------
[verify] > No Errors or Warnings
----- output end -----
[smoke] > quarto render docs/smoke-all/2023/01/04/issue-3847.qmd --to html ... ok (320ms)
ok | 1 passed | 0 failed (1s)
> Exiting virtualenv activated for tests
# run tests for all documents in docs/smoke-all/
./run-tests.ps1 smoke/smoke-all.tests.ts
# run tests for some `.qmd` document in a specific place (using glob)
./run-tests.ps1 docs/smoke-all/2022/**/*.qmd
# Or using longer version
./run-tests.ps1 smoke/smoke-all.test.ts -- docs/smoke-all/2022/**/*.qmd
# run test for a specific document
./run-tests.ps1 docs/smoke-all/2023/01/04/issue-3847.qmd
# Or using longer version
./run-tests.ps1 smoke/smoke-all.test.ts -- docs/smoke-all/2023/01/04/issue-3847.qmd
Examples of tests output after it ran
./run-tests.ps1 smoke/smoke-all.test.ts -- docs/smoke-all/2023/01/04/issue-3847.qmd
> Setting all the paths required...
> Checking and configuring environment for tests
>>>> Configuring R environment
* The library is already synchronized with the lockfile.
>>>> Configuring python environment
Setting up python environnement with pipenv
Installing dependencies from Pipfile.lock (0ded54)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
>>>> Configuring Julia environment
Setting up Julia environment
Building Conda ─→ `C:\Users\chris\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\e32a90da027ca45d84678b826fffd3110bb3fc90\build.log`
Building IJulia → `C:\Users\chris\.julia\scratchspaces\44cfe95a-1eb2-52ea-b672-e2afdf69b78f\59e19713542dd9dd02f31d59edbada69530d6a14\build.log`
>>>> Configuring TinyTeX environment
tinytex is already installed and up to date.
> Preparing running tests...
> Activating virtualenv for Python tests
> Running tests with "C:\Users\chris\Documents\DEV_R\quarto-cli\package\dist\bin\tools\deno.exe test --config test-conf.json --unstable-ffi --allow-read --allow-write --allow-run --allow-env --allow-net --check --importmap=C:\Users\chris\Documents\DEV_R\quarto-cli\src\dev_import_map.json smoke/smoke-all.test.ts -- docs/smoke-all/2023/01/04/issue-3847.qmd"
running 1 test from ./smoke/smoke-all.test.ts
[smoke] > quarto render docs\smoke-all\2023\01\04\issue-3847.qmd --to html ...
------- output -------
[verify] > No Errors or Warnings
----- output end -----
[smoke] > quarto render docs\smoke-all\2023\01\04\issue-3847.qmd --to html ... ok (650ms)
ok | 1 passed | 0 failed (2s)
-
smoke-all.test.ts
accept only one argument. You need to use glob pattern to run several smoke-all test documents. -
Individual
smoke-all
tests and other test can't be run at the same time withrun-test.[sh|ps1]
. This is becausesmoke-all.test.ts
requires arguments. If a smoke-all document and another smoke-test are passed as argument, the smoke-all test will be prioritize and other will be ignored (with a warning).
Example with Linux:
You can do
# run all smoke-all tests and another smoke test
./run-tests.sh smoke/extensions/extension-render-doc.test.ts smoke/smoke-all.test.ts
# run tests for some `.qmd` document in a specific place (using glob)
./run-tests.sh docs/smoke-all/2022/**/*.qmd
Don't do
# run .qmd smoke-all test and another smoke test - smoke-all test will have the priority and other will be ignored (with a warning)
./run-test.sh smoke/extensions/extension-render-doc.test.ts ./docs/smoke-all/2023/01/04/issue-3847.qmd
# run smoke-all.test.ts with argument and another smoke test
./run-tests.sh smoke/extensions/extension-render-doc.test.ts smoke/smoke-all.test.ts -- ./docs/smoke-all/2023/01/04/issue-3847.qmd
.vscode/launch.json
has a Run Quarto test
configuration that can be used to debug when running tests. One need to modify the program
and args
fields to match the test to run.
Example:
"program": "smoke/smoke-all.test.ts", // test script here
"args": ["--", "tests/docs/smoke-all/2023/01/04/issue-3847.qmd"], // args to the script here, like in command line smoke/smoke-all.test.t -- .\docs\smoke-all\2023\01\19\2107.qmd
Short version can't be use here as we are calling deno test
directly and not run-tests.sh
script.
Linux only
This lives in run-parallel-tests.ts
and called through run-parallel-tests.sh
.
- It requires a text file with tested timed and following a specific format. (Default is
timing.txt
and here is an example in our repo) - Based on this file, the tests will be split in buckets to minimize the tests time (buckets are filled by their minimum overall time).
- Then
./run-tests.sh
will be run for each bucket from deno usingPromise.all()
andrun-tests.sh
on the whole bucket's test files, so that the buckets are ran in parallel.
This is a simple way to run all the tests or a subset of tests in parallel locally.
To create a timed test file like timing.txt
, this command needs to be run.
QUARTO_TEST_TIMING='timing.txt' ./run-tests.sh
When this is done, any other argument will be ignored, and the following happens
- All the
*.test.ts
file are found and run individually using/usr/bin/time
to store timing in the file - When
smoke-all.test.ts
is found, all the*.qmd
,*.md
and*.ipynb
indocs/smoke-all/
not starting with_
are found and run individually using same logic. This means eachsmoke-all
test is timed.
The results is written in the $QUARTO_TEST_TIMING
file. Here is an example:
./smoke/directives/include-fixups.test.ts
0.02 real 0.02 user 0.00 sys
./smoke/filters/filters.test.ts
3.26 real 3.79 user 0.47 sys
./smoke/filters/editor-support.test.ts
0.72 real 0.58 user 0.14 sys
./smoke/engine/include-engine-detection.test.ts
3.61 real 3.11 user 0.24 sys
./smoke/smoke-all.test.ts -- docs/smoke-all/2022/12/12/code-annotation.qmd
4.81 real 4.32 user 0.33 sys
./smoke/smoke-all.test.ts -- docs/smoke-all/2022/12/9/jats/computations.out.ipynb
2.22 real 2.83 user 0.27 sys
This will be read by run-parallel-tests.ts
to get the real
value and fill the bucket based on it.
smoke-all
tests are special because they are in the form of individual .qmd
or .ipynb
document that needs to be run using smoke-all.test.ts
script, with arguments. Unfortunately, this prevent running individual smoke-all
documents in same buclets as other individual smoke test (which are their own .test.ts
file).
So, if the timed file contains some individual timing for smoke-all
documents like this
./smoke/smoke-all.test.ts -- docs/smoke-all/2022/12/12/code-annotation.qmd
then they are ignored and .smoke-all.test.ts
will be run in its own bucket. It will usually be the longest test run.
Individual smoke-all
tests timing are useful for Quarto parallelized smoke tests on GHA CI as the buckets are split into their own runners and each test in a bucket if run using run-test.sh
. This allows a bucket to contains some *.test.ts
but also some document *.qmd
or *.ipynb
. More details in test-smoke.yml and test-smokes-parallel.yml
-n=
: Number of buckets to create to run in parallel.run-parallel-tests.sh -n=5
split tests in 5 buckets and run them at the same time. For local run,n
should be a number of core. For CI run,n
will be the number of runners to use at the same time (mulplied by 2 because Linux and Windows are ran on CI).--verbose
: show some verbosity. Otherwise, no specific logging in console in done.--dry-run
: show the buckets of tests, but do not run. Otherwise, they are run.--timing-file=
: Which file to use as timed tests information to creates the buckets. (default totiming.txt
).run-parallel-tests.sh --timing-file='timing2.txt'
will usetiming2.txt
to run the file.--json-for-ci
: Special flag to trigger splitting tests in buckets for the parallel run on CI and that makesrun-parallel-tests.sh
outputs JSON string specifically formatted for GHA processing.
test-smokes-parallel.yml
will be triggered to loadtiming-for-ci.txt
and split tests in buckets. It will create a matrix to triggertest-smokes.yml
onworkflow_call
event for each bucket.- PR against main and commits to main will trigger this workflow, and tests will be ran in parallel jobs.
- A
workflow_dispatch
event can be used to trigger it through API call,gh
CLI tool or GHA GUI online.
test-smokes.yml
is the main CI workflow which configure the environment, and run the tests on Ubuntu and Windows.- If it was triggerred by
workflow_call
, then it will run each test in usingrun-tests.[sh|ps1]
in a for-loop. - Scheduled tests are still run daily in their sequential version.
- If it was triggerred by