diff --git a/docs/computing/running/throughput.md b/docs/computing/running/throughput.md index e2f06942e0..60be7aecde 100644 --- a/docs/computing/running/throughput.md +++ b/docs/computing/running/throughput.md @@ -58,8 +58,8 @@ graph TD E -->|Serial| F(GNU Parallel
Array jobs
HyperQueue) E -->|Parallel| G(Single- or multinode subtasks?) G -->|Single| H(Dependencies between subtasks?) - G -->|Multi| I(FireWorks) - H -->|Yes| J(Snakemake
Nextflow
FireWorks) + G -->|Multi| I(FireWorks
Toil-CWL-Runner) + H -->|Yes| J(Snakemake
Nextflow
FireWorks
Toil-CWL-Runner) H -->|No| K(HyperQueue) ``` @@ -157,6 +157,8 @@ graph TD with `xargs`, see [xargsjob.sh] for example. * [FireWorks] is a flexible tool for defining, managing and executing workflows with multiple steps and complex dependencies +* [Toil-CWL-Runner] is an open source workflow manager using the Common + Workflow Language open standards * [HyperQueue] is a tool for efficient sub-node task scheduling * [Nextflow workflows using HyperQueue as an executor] can be leveraged to run large workflows involving thousands of processes efficiently @@ -199,6 +201,7 @@ workflows. [HyperQueue]: ../../apps/hyperqueue.md [GNU Parallel]: ../../support/tutorials/many.md [FireWorks]: fireworks.md +[Toil-CWL-Runner]: toil-cwl-runner.md [contact CSC Service Desk]: ../../support/contact.md [Nextflow]: ../../support/tutorials/nextflow-puhti.md [Snakemake]: https://snakemake.readthedocs.io/en/stable/ diff --git a/docs/computing/running/toil-cwl-runner.md b/docs/computing/running/toil-cwl-runner.md new file mode 100644 index 0000000000..72271de363 --- /dev/null +++ b/docs/computing/running/toil-cwl-runner.md @@ -0,0 +1,121 @@ +# Running CWL workflows on Puhti with `toil-cwl-runner` + +![CWL Logo](https://raw.githubusercontent.com/common-workflow-language/cwl-website/main/content/assets/img/CWL-Logo-HD-cropped2.png){ width=50% } + +The [Common Workflow Language](https://www.commonwl.org/) is a popular set of open standards implemented by several workflow runners and platforms. +The CWL standards are targeted at creating portable workflows made of command line programs. The steps can be written in any compiled or interpreted language. +Sub-workflows, optional steps, scatter-gather, and implicit parallelism are just some of the features. + +The [Toil workflow system](https://toil.ucsc-cgl.org/) supports running CWL on a variety of schedulers and systems. + +This page describes how run CWL worklflows on Puhti using `toil-cwl-runner`, including the usage of `apptainer` to execute any provided Docker-format containers. + +## Strengths of the Common Workflow Language standards + +- Open standard (free to read, free to contribute to) governed by a [not-for-profit charity which is legally obligated to work in the public interest](https://sfconservancy.org/news/2018/apr/11/cwl-new-member-project/). +- [Multiple implementations](https://www.commonwl.org/implementations/) of the CWL standards +- Used in many [different fields of research](https://www.commonwl.org/gallery/) +- YAML based syntax with [special support in many IDEs](https://www.commonwl.org/tools/#editors) +- Support, but does not require, software containers. Can also work with conda packages, `module load` environments, and locally available software. +- CWL's model works hard to keep site-specific deatuls out of the workflow definition. Enabling portability between laptops, clusters, and cloud systems. + +## Strengths of `toil-cwl-runner` +- Supports sending jobs to Slurm, translating CWL resource requirements to Slurm resources specifications. +- Even when using Slurm, (sub-)tasks do not have to have identical resource requirements. +- Can also run on other batch systems: Grid Engine, Torque, LSF, HTCondor. +- Launches and monitors Slurm jobs for you. Also constructs the `apptainer` commands (or some other software container engine as appropriate: `docker`, `podman`, `singularity`, `udocker`). +- No database needs to be setup. + +## Disadvantages for using `toil-cwl-runner` +- Just a workflow runner. Won't manage your data, or keep track of previous workflow runs. + +## Installing `toil-cwl-runner` + +Install `toil` with `CWL` plugin. +``` +module load python-data + +cd /projappl/ +python -m venv venv +source venv/bin/activate + +pip install -U setuptools wheel +pip install toil[cwl] + +toil-cwl-runner --version +``` + +Install `nodejs` which provides helpful tools for debugging `toil` internals. +``` +cd /projappl/project_nnnnnnn +wget https://nodejs.org/dist/v18.16.1/node-v18.16.1-linux-x64.tar.xz +tar -xf node-v18.16.1-linux-x64.tar.xz +export PATH=$PATH:/projappl/project_nnnnnnn/node-v18.16.1-linux-x64/bin +``` + +## Defining CWL workflows + +Learning resources + +- [Novice CWL tutorial](https://carpentries-incubator.github.io/cwl-novice-tutorial/), includes detailed setup instructions for local editing and running on Microsoft Windows, macOS, and Linux +- + +## Running CWL workflows with `toil-cwl-runner` + +!!! Note + Singularity containers can't be run in the **login node** or in an **interactive session** due to network constraints. + +When you have defined a workflow with `CWL`, you can send it to the cluster using `sbatch`, and then `toil` will start new jobs for each item in the workflow description. + +### Preliminary Steps +Create working directories for `toil`. +``` +mkdir /projappl/project_nnnnnnn/ +mkdir /projappl/project_nnnnnnn/ +mkdir /projappl/project_nnnnnnn/ +``` + +### Creating the sbatch file +The `sbatch` file `workflow.sh` will reference the `CWL` file `workflow.cwl` where you have described your workflow steps. + +!!! Note + See [batch documentation](./creating-job-scripts-puhti.md) on how to fill out the `#SBATCH` values. + +`workflow.sh` +```bash +#!/bin/sh +#SBATCH --job-name= +#SBATCH --account= +#SBATCH --time=01:00:00 +#SBATCH --mem-per-cpu=1G +#SBATCH --nodes=1 +#SBATCH --cpus-per-task=2 +#SBATCH --partition=small + +source /projappl//venv/bin/activate + +WORKDIR=/projappl/project_nnnnnnn +SCRATCH=/scratch/project_nnnnnnn +export TMPDIR=$WORKDIR/tmp +export TOIL_SLURM_ARGS="--account=project_nnnnnnn --partition=small" +export CWL_SINGULARITY_CACHE=$WORKDIR/singularity +unset XDG_RUNTIME_DIR + +TOIL_SLURM_ARGS="--account= --partition=small" toil-cwl-runner \ + --jobStore $WORKDIR/ \ + --workDir $SCRATCH/ \ + --tmpdir-prefix $TMPDIR/ \ + --batchSystem slurm \ + $WORKDIR/workflow.cwl \ + --message "message for job" +``` + +Send your workflow to the cluster. +``` +sbatch workflow.sh +``` + +## Monitoring a running workflow + +Check the output logs from the main Toil job or +run `toil status $WORKDIR/`. diff --git a/docs/support/tutorials/index.md b/docs/support/tutorials/index.md index 18cdc45c01..a801503e01 100644 --- a/docs/support/tutorials/index.md +++ b/docs/support/tutorials/index.md @@ -24,6 +24,7 @@ * [General high-throughput guidelines](../../computing/running/throughput.md) * [Running Nextflow workflows using HyperQueue](nextflow-hq.md) * [FireWorks workflow manager](../../computing/running/fireworks.md) +* [Toil CWL workflow manager](../../computing/running/toil-cwl-runner.md) * [How to run many short jobs with GNU Parallel](many.md) * [HyperQueue meta-scheduler](../../apps/hyperqueue.md)