Add copyFiles pipeline

[bilalebi]

JIRA Ticket

https://www.ebi.ac.uk/panda/jira/browse/EA-1276

Description

This pipeline copies files from a source directory to a destination using rsync. It optionally sends email notifications on completion or failure.

Use case & Benefits

With this pipeline, we will be able to automatically copy files from one directory to another

nextflow run copyFiles.nf -c ../filescopy.config --source /path/to/source/data  --destination /path/to/destination --file_format .vcf --send_email yes --email_recipient [email protected]

Features

• File Transfer: Uses rsync to copy files from the specified source to the destination.
• File Filtering: Only processes files matching the specified format (e.g., .vcf).
• Optional Email Notifications: Sends an email with the pipeline's status, controlled by a send_email flag.
• Directory Checks: Ensures source and destination directories exist before execution.

Testing

I tested the pipeline locally and it's working as expected

• Test it on codon

[dpopleton]

It is fine, baring the comments, but not too useful at the moment.
I am approving, but we need some sort of DC, even a dummy, before I want to include it in pipelines. I am not sure how you want to do that with a full directory sync, as opposed to single files.
I thought something like this:

process InitialDatacheck {
    label 'mem2GB_DM'
    input:
    path initial_file

    output:
    path initial_file

    script:
    """
    echo "Running datacheck on initial file ${initial_file}"
    ensembl-datacheck --file ${initial_file} --test=${params.file_type}
    """
}

But that can come later, if need be.

[dpopleton]

We need the slurm configuration from production.config

[nwillhoft]

Could we simplify this by using the built-in Nextflow workflow notification report with -N? e.g. https://www.nextflow.io/docs/latest/notifications.html#workflow-notification

[bilalebi]

The reason why I though of creating a custom email is because -N doesn't mention the source and destination locations.

But if it's not needed, we can use -N feature as you suggested