Transfer files from DWD HTTPs server to Source Co-Op using `rclone copyurl --urls` by JackKelly · Pull Request #406 · dynamical-org/reformatters

JackKelly · 2026-01-31T11:41:46Z

This PR should be ready for testing in production. I've tested it on an EC2 VM, copying to an S3 bucket, and it was getting 50 MiB/s, and it only takes about 30 secs to check an entire NWP run 🙂

Aims

Speed up every part of the file transfer from DWD to Source Co-Op.
Provide a solution that can handle DWD's planned directory format.

Implementation notes

The approach in this PR differs from the previous approach (PR #360 & #370) in that this PR:

Downloads files from DWD's HTTPS server instead of the FTP server.
- DWD's FTP server doesn't allow more than 10 concurrent connections from one IP address. This lack of concurrency was killing transfer performance given the ~150ms latency over the 8,000 km from Germany to us-west-2, and given the tiny size of the grib files, and given the "chatty" nature of the FTP protocol. FTP requires about 7 round trips per file, which, over 8,000 km, is 1 second of overhead per file, even though the actual payload might only take a fraction of a second to transfer. In contrast, DWD's HTTPS server doesn't appear to have an upper limit on concurrent connections. And HTTP is less "chatty" than FTP: once the connection is established, HTTPS only needs 1 back-and-forth per file.
Uses rclone copyurl --urls src_and_dst_paths.csv instead of rclone copy.
- rclone copyurl is a gem hidden deep within rclone's docs, and I only discovered it by chance a few days ago (and the docs don't make it clear that copyurl is capable of solving our problems, to the extent that Gemini firmly believes that copyurl cannot do what this PR does with copyurl! (UPDATE: I've submitted a PR to rclone to update the docs)). AFAICT, copyurl is the only rclone command that allows us to provide a list of mappings from full source URL to full destination path. In contrast, rclone copy --files-from only allows us to provide a list of source files; rclone copy can't modify the structure of the destination path (e.g. extracting the NWP init datetime from the base filename of the grib, and using that datetime in an early part of the destination path). rclone copyurl is wonderful because it allows us to provide a list of arbitrary mappings from source URL to destination path.
Uses Python to figures out which files have already been transferred
- The previous design relied on rclone copy to figure out which files already exist on the destination. This was slow.
- In contrast rclone copyurl doesn't check the destination before copying. It just blindly copies.
- So, the Python code in this PR first asks rclone to list all the files on the HTTPS server, and list the relevant files in the destination path, and then our Python code computes the set difference. So we can guarantee that this check only happens once per run. And it's fast. It only takes about 30 seconds to check an entire NWP run, whereas the old approach took several minutes to check the larger NWP variable directories.
Simplified logging
- Instead of writing fiddly Python code to parse JSON logs from rclone, this PR instead takes a simpler approach: We just persuade rclone to output useful logs as strings, and log those strings. No more JSON parsing; and no more adding up TransferSummary objects. We still get a regular log saying how much data has been transferred, and the throughput. We do lose a tiny bit of functionality, in that the current version doesn't say how much is left to transfer. We can add that back in if needs be.

TODO

Put into a src/reformatters/dwd/archive_gribs folder, with an __init__.py.
Split into several Python files.
Benchmark on an OCF AWS EC2 VM, copying from DWD HTTPS to S3. DONE. It's fast! (~50 MiB/s). Logs here.
Write unit tests

This PR closes: Speed up rclone copy from DWD's FTP server to Source Co-Op #392
My raw research notes: Research into speeding up rclone copy from DWD's FTP server to Source Co-Op #390
More research notes (in Google docs)
Previous approach PR 2/2: Production code to copy ICON-EU grib files from FTP to Source Co-Op #370
Previous approach PR 1/2: Copy files from DWD FTP using rclone #360
Save ICON-EU GRIB files from DWD's FTP server to cloud object storage #258
DWD ICON-EU forecast task list & general discussion (for live-updating Zarr) #173

found out about copyurls, which is much better.

of the heavy lifting.

little.

aldenks · 2026-02-02T04:20:54Z

🎉 Will review soon!

JackKelly · 2026-02-02T08:29:17Z

Thank you! No huge rush - I'll be busy this week with other stuff. And sorry for this being the third re-write of this functionality! "Third time's a charm", right?! I'm feeling optimistic that this approach is the simplest yet 🙂

JackKelly · 2026-02-02T10:14:27Z

src/reformatters/dwd/archive_gribs/copy_files_from_dwd.py

+    for src_path in src_paths_starting_with_nwp_var:
+        dst_path = convert_src_path_to_dst_path(src_path)
+        if dst_path not in files_already_on_dst:
+            full_src_path = f"{src_host_and_root_path}/{src_path}"


I'd be tempted to use urlpath.URL here, to ensure we join URLs correctly, if you're happy for me to add urlpath as a dependency?

would the benefit be to make sure we don't have double slashes? i think id rather ensure that with a unit test or assert since we don't have many cases to deal with here if that's the specific benefit, but maybe there's more?

aldenks

Awesome! Thanks for the iterations on this -- "easy" things often aren't so easy at some scale 🙃

If you want to merge and do follow ups in a separate PR thats good too, they are all minor. I'll let you know when i can get this up and testing, not sure exactly but in the next week for sure.

aldenks · 2026-02-03T19:20:55Z

src/reformatters/dwd/archive_gribs/copy_files_from_dwd.py

+    for src_path in src_paths_starting_with_nwp_var:
+        dst_path = convert_src_path_to_dst_path(src_path)
+        if dst_path not in files_already_on_dst:
+            full_src_path = f"{src_host_and_root_path}/{src_path}"


would the benefit be to make sure we don't have double slashes? i think id rather ensure that with a unit test or assert since we don't have many cases to deal with here if that's the specific benefit, but maybe there's more?

aldenks · 2026-02-03T19:24:38Z

src/reformatters/dwd/archive_gribs/list_files.py

+            _log_error_from_called_process_error(e)
+            raise


what about a

try:
_log_err...
finally:
raise

to make sure the nice logging doesn't get in the way of raising?

aldenks · 2026-02-03T19:28:24Z

src/reformatters/dwd/archive_gribs/path_conversion.py

+def extract_nwp_init_datetime_from_grib_filename(
+    grib_filename: str,
+) -> datetime:
+    dwd_nwp_init_date_regex: Final[re.Pattern[str]] = re.compile(r"_(\d{10})(?=_)")


its probably not material perf wise, but the .compile is happening on each call. We could put in a constant

aldenks · 2026-02-03T20:12:27Z

src/reformatters/dwd/icon_eu/forecast/dynamical_dataset.py

        # It would've made more sense for `dst_root` to be a `PurePosixPath` but Typer doesn't
        # handle `PurePosixPath`, so we use a `str` to keep Typer happy.
-        dst_root: str = grib_archive_path,
+        dst_root_path: str = grib_archive_path,


Will you make the grib_archive_path Final? and maybe it should be called dynamical_grib_archive_rclone_root to make it clear its our archive and rclone specific

aldenks · 2026-02-03T20:26:36Z

Actually, @JackKelly im going to merge this so i can wrap up the ty switch

JackKelly · 2026-02-03T20:36:44Z

Awesome, thanks! I'll implement your suggestions in a follow-up PR!

JackKelly added 5 commits January 30, 2026 21:44

It's working well. Live async logging isn't going so well.

17f923a

Nice, it's now outputting JSON stats to stderr

c10e968

Adding my old attempt at using rclone remote control daemon, before I

c8347da

found out about copyurls, which is much better.

Enable stats-log-level=NOTICE

7235c9a

Merge branch 'main' into rclone-https

54a1768

JackKelly self-assigned this Jan 31, 2026

JackKelly added the enhancement New feature or request label Jan 31, 2026

JackKelly requested review from aldenks and removed request for aldenks January 31, 2026 11:42

JackKelly added 13 commits January 31, 2026 13:24

Add lots of docstrings, and tidy

a75d813

Handle missing directory. And other improvements.

643e1f3

outputs stats only direct from rclone

34119a7

Logging stats works nicely now. Very simple. Just get rclone to do most

041dbea

of the heavy lifting.

Pass env_vars, checkers, and transfers throughout code. Refactor a

df00c9a

little.

Re-order functions to be roughly in the order they are called

bd120af

Move functions into new Python modules. Integrate with DynamicalDataset

261d1b2

Delete old tests

ce769bd

Fix template_config_test for ICON-EU simply by importing rioxarray

2a7ea3e

Merge branch 'fix-dwd-template-config-test' into rclone-https

a6d4229

Fix typo in comments

90d4465

Gemini has written some tests. And tweaked the code under test.

f3b0605

Tweak wording in test_rclone_copyurl

34477c6

JackKelly marked this pull request as ready for review January 31, 2026 22:08

JackKelly requested a review from aldenks February 2, 2026 08:27

JackKelly commented Feb 2, 2026

View reviewed changes

Merge branch 'main' into rclone-https

7a403d3

aldenks mentioned this pull request Feb 2, 2026

Replace mypy with ty for type checking #400

Merged

aldenks approved these changes Feb 3, 2026

View reviewed changes

aldenks merged commit 38e4ecf into dynamical-org:main Feb 3, 2026
2 checks passed

JackKelly deleted the rclone-https branch February 3, 2026 20:37

JackKelly mentioned this pull request Feb 4, 2026

DWD ICON-EU forecast task list & general discussion (for live-updating Zarr) #173

Open

28 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transfer files from DWD HTTPs server to Source Co-Op using `rclone copyurl --urls`#406

Transfer files from DWD HTTPs server to Source Co-Op using `rclone copyurl --urls`#406
aldenks merged 19 commits intodynamical-org:mainfrom
JackKelly:rclone-https

JackKelly commented Jan 31, 2026 •

edited

Loading

Uh oh!

aldenks commented Feb 2, 2026

Uh oh!

JackKelly commented Feb 2, 2026

Uh oh!

JackKelly Feb 2, 2026

Uh oh!

aldenks Feb 3, 2026

Uh oh!

aldenks left a comment

Uh oh!

aldenks Feb 3, 2026

Uh oh!

aldenks Feb 3, 2026

Uh oh!

aldenks Feb 3, 2026

Uh oh!

aldenks Feb 3, 2026

Uh oh!

aldenks commented Feb 3, 2026

Uh oh!

Uh oh!

JackKelly commented Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

JackKelly commented Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Aims

Implementation notes

TODO

Related

Uh oh!

aldenks commented Feb 2, 2026

Uh oh!

JackKelly commented Feb 2, 2026

Uh oh!

JackKelly Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks left a comment

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

aldenks commented Feb 3, 2026

Uh oh!

Uh oh!

JackKelly commented Feb 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JackKelly commented Jan 31, 2026 •

edited

Loading