Feature request: per-process cap on “in-flight outputs”

[asuq]

New feature

Hello Nextflow maintainers,
thank you very much for developing Nextflow and for the continuous effort in maintaining it.

I would like to propose a new process-level control to prevent large intermediate outputs from accumulating in work/ when an upstream process is much faster than the downstream one.

In short: a per-process cap on “in-flight outputs” (outputs already produced by a process but not yet fully consumed/drained by downstream), to provide backpressure and bound peak disk usage.

Use case

Main scenario is HPC (or any quota-limited environment) where work/ is on a filesystem with strict disk quota.

Typical pattern:

• Process A: fast, produces very large output per sample (e.g. 10–100 GB)
• Process B: slow (or low parallelism), consumes that output
• With ~100 samples, Process A can finish quickly and produce many huge outputs, while Process B is still processing only a few.
• These outputs must stay in work/ until downstream is finished, so temporary disk usage grows a lot and can exceed quota.

I also use nf-boost to minimise work/, but it cannot remove those intermediates early because they are still required by downstream tasks. So the peak usage problem still happens.

Why existing feature do not solve it:

• executor.queueSize is global; it does not enforce fairness between processes (fast upstream can fill it first).
• Reducing upstream maxForks is possible but often requires manual tuning and can reduce throughput; also it does not address submission starvation in a general way.
• nf-boost helps only when files are no longer needed; it cannot prevent accumulation when downstream is slow.

Related discussions:

• Yet another "huge intermediate files" issue #2468 (huge intermediate files; merging processes often suggested but not always desirable)
• Remove single processed files before the end of the whole process #1754 (ideas about earlier cleanup / downstream start; same pain point but different direction)

Suggested implementation

Add a per-process directive (name is flexible) that limits how many outputs from that process can be “in flight” at the same time.

Example (illustrative):

  process {
    withName: PROCESS_A {
      maxInFlight = 5
    }
  }

Proposed semantics:

• For a process with maxInFlight = N, Nextflow should allow at most N tasks worth of outputs to be produced and remain “pending” for downstream.
• If the limit is reached, new tasks of that process should not start / not be submitted, until downstream has progressed enough that some previously produced outputs are no longer pending (e.g. all dependent downstream tasks for those outputs have completed successfully).
• This is not asking to delete files unsafely: it is only restricting ahead-of-time production to keep disk usage bounded.

Implementation building blocks:

• Maintain an “in-flight counter” per process (or per output channel edge), increment when a task of that process completes and materialises its outputs.
• Decrement when all downstream tasks that consume that output are completed (for simple linear A→B it is just when the corresponding B task finishes; for branching, when all dependants finish).
• Integrate this check into the task dispatch / submission logic so that producers can be paused when they outrun consumers.

Environment:

• Nextflow version: v25.10.2
• Executor: slurm
• nf-boost version: v0.6.0

Thanks again for your time.

[bentsherman]

Check out the maxForks directive

[asuq]

Hi @bentsherman,
Thanks for the suggestion.

I did check and try the maxForks directive, but it doesn’t completely solve this specific problem.
(Also, I previously used the word "intermediates" loosely, here I mean the output of Process A)

maxForks limits how many tasks of Process A run at the same time, but the disk issue is mainly about how many completed outputs from Process A are sitting on disk while waiting for the slow Process B.

Because Process A is fast, it can keep cycling through the limited fork slots and still finish far ahead of Process B.
The result is that a large number of A’s big outputs can accumulate in work/ and remain there until B finishes consuming them.
This happens even when using nf-boost cleanup, because A’s outputs cannot be removed early while they are still required by Process B.

In practice, to stay under quota I would need to set maxForks extremely low (sometimes 1–2), which reduces throughput a lot and requires manual tuning per dataset/run.

So my request is slightly different from concurrency limiting: I’m looking for a process-level backpressure control, e.g. a cap on “in-flight outputs” (completed outputs produced by A but not yet drained/consumed by downstream).
This would bound peak disk usage in a more direct and robust way than maxForks, especially when used with nf-boost cleanup.

[bentsherman]

Ah you're right, there is a difference between in-flight tasks and in-flight "outputs"

Actually I've been interested in this problem for a long time because we had the exact same problem with GEMmaker and generating too many BAM outputs at once. You might be interested in the hacks they came up with to work around this problem

Automatic cleanup (#452) is part of the solution, but not enough on its own since the garbage collector can't delete the intermediate outputs until the downstream tasks have consumed them

I've always focused on the approach of merging processes into one. I'd like to find a way to do this automatically (see #2527 ) so that you can still keep things modular. It solves the problem indirectly by ensuring the intermediate output is immediately consumed, especially if you use scratch storage. There may be other ways to do it... you just need some way to mark a group of processes as a single unit