Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download ECCOMetadata in parallel? #201

Open
glwagner opened this issue Oct 25, 2024 · 0 comments
Open

Download ECCOMetadata in parallel? #201

glwagner opened this issue Oct 25, 2024 · 0 comments
Labels
data wrangling We must feed the models so they don't get cranky

Comments

@glwagner
Copy link
Member

I think it might speed things up rather than doing the downloads in serial:

for data in metadata
filename = metadata_filename(data)
shortname = short_name(data)
if !isfile(filename)
msg = "\n See ClimaOcean.jl/src/ECCO/README.md for instructions."
isnothing(username) && throw(ArgumentError("Could not find the username for $(url). Please provide a username in the ECCO_USERNAME environment variable." * msg))
isnothing(password) && throw(ArgumentError("Could not find the username for $(url). Please provide a password in the ECCO_PASSWORD environment variable." * msg))
# Version specific download file url
if data.version isa ECCO2Monthly || data.version isa ECCO2Daily
fileurl = joinpath(url, shortname, filename)
elseif data.version isa ECCO4Monthly
year = string(Dates.year(data.dates))
fileurl = joinpath(url, shortname, year, filename)
end
cmd = `wget --http-user=$(username) --http-passwd=$(password) --directory-prefix=$(path) $(fileurl)`
run(cmd)
end
end

@glwagner glwagner added the data wrangling We must feed the models so they don't get cranky label Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data wrangling We must feed the models so they don't get cranky
Projects
None yet
Development

No branches or pull requests

1 participant