Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAG creation extremely slow with storage function targeting zip files #25

Open
FabianHofmann opened this issue May 27, 2024 · 0 comments

Comments

@FabianHofmann
Copy link

The storage function can lead to very long DAG creation times when it is pointing to online zip files.

The following example shows it quite clearly.

Snakefile:

rule retrieve_eurostat_data:
    input:
        storage(
            "https://ec.europa.eu/eurostat/documents/38154/4956218/Balances-April2023.zip", 
        ),

When running snakemake -n, the DAG creation takes longer than two minutes (direct download time via browser ~20 seconds)

I don't know whether it is related to the fact, that snakemake runs the download multiple times even though it is in dry-run mode?

Let me know if there is a way to support or if you need more information/context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant