Azure Batch executor fails if `.command` files are included in the output

[adamrtalbot]

Bug report

If the .command.* files are included in the output syntax, a pipeline executed on Azure Batch will fail.

Expected behavior and actual behavior

The pipeline should upload the .command.* files successfully and use them as outputs. It could do this by de-duplicating the output files

It's possible this is caused by the output file is uploaded once and then uploaded a second time. Alternatively it could be caused by parsing the nxf_az_upload paths, which may have a syntax error caused by the preceeding .. Not clear on the root cause.

Steps to reproduce the problem

main.nf:

process HELLO {

    container 'ubuntu:22.04'

    output:
        path "hello.txt"
        path(".command.*", hidden: true)

    script:
    """
    echo 'hello!' > hello.txt
    """
}

workflow {
    HELLO()
}

Program output

>  nextflow run . -c azure.config
N E X T F L O W  ~  version 25.04.6
Launching `./main.nf` [lonely_mahavira] DSL2 - revision: cc808808e6
[ca/d27186] Submitted process > HELLO
ERROR ~ Error executing process > 'HELLO'

Caused by:
  Process `HELLO` terminated with an error exit status (1)


Command executed:

  echo 'hello!' > hello.txt

Command exit status:
  1

Command output:
  Number of Folder Property Transfers: 0
  Total Number of Transfers: 1
  Number of Transfers Completed: 0
  Number of Transfers Failed: 1
  Number of Transfers Skipped: 0
  TotalBytesTransferred: 0
  Final Job Status: Failed
  
  
  
  
  Job 6b2c5f00-f1cf-fe4f-59fe-08508991a7e8 summary
  Elapsed Time (Minutes): 0.0336
  Number of File Transfers: 1
  Number of Folder Property Transfers: 0
  Total Number of Transfers: 1
  Number of Transfers Completed: 1
  Number of Transfers Failed: 0
  Number of Transfers Skipped: 0
  TotalBytesTransferred: 6
  Final Job Status: Completed
  
  
  
  
  Job cb8e5077-a935-ee46-6d44-043b5b8a94fc summary
  Elapsed Time (Minutes): 0.0335
  Number of File Transfers: 1
  Number of Folder Property Transfers: 0
  Total Number of Transfers: 1
  Number of Transfers Completed: 0
  Number of Transfers Failed: 1
  Number of Transfers Skipped: 0
  TotalBytesTransferred: 0
  Final Job Status: Failed
  
  
  
  
  Job 949cbf6c-dee9-ee4d-6e8b-b3696248c0da summary
  Elapsed Time (Minutes): 0.034
  Number of File Transfers: 1
  Number of Folder Property Transfers: 0
  Total Number of Transfers: 1
  Number of Transfers Completed: 1
  Number of Transfers Failed: 0
  Number of Transfers Skipped: 0
  TotalBytesTransferred: 0
  Final Job Status: Completed

Work dir:
  az://storageaccount/scratch/ca/d2718644ae026aa4c08c2bc5c19c0d

Container:
  ubuntu:22.04

Environment

• Nextflow version: [?]
• Java version: [?]
• Operating system: [macOS, Linux, etc]
• Bash version: (use the command $SHELL --version)

Additional context

(Add any other context about the problem here)

[bentsherman]

@jorgee I haven't looked at this too closely but I wonder if this is related to the other issue you are addressing in #6364

[adamrtalbot]

Note as a workaround...don't include the .command.* files. It's not necessary anyway.

[jorgee]

It seems that the .command.out is modified during the transfer. Checking at the az copy logs, I got this error:

2025/09/03 14:10:31 ERR: [P#0-T#0] UPLOADFAILED: /mnt/batch/tasks/workitems/job-9dd0ad5b49fb879fd7fd-HELLO/job-1/nf-d5468edb9db0865f9ec52d253419c4a7/wd/.command.out : 000 : source modified during transfer.

Looking at the .command.run, I see the unstage is redirecting the outputs to .command.out and .command.err. This could be the cause of the issue why copying .command.out is failing

nxf_unstage() {
    if [[ ${nxf_main_ret:=0} == 0 ]]; then
        (set -e -o pipefail; (nxf_unstage_outputs | tee -a .command.out) 3>&1 1>&2 2>&3 | tee -a .command.err)
        nxf_unstage_ret=$?
    fi
    nxf_unstage_controls
}