Add guidance for CO HDF/NetCDF

[abarciauskas-bgse]

Adds long overdue and much requested guidance on cloud-optimizing HDF(5) and NetCDF(-4).

I've added as co-authors @ajelenak and @ashiklom but also cited @bilts @betolink and @andypbarrett, so tagging all for review.

[github-actions]

PR Preview Action v1.4.8
🚀 Deployed preview to https://cloudnativegeo.github.io/cloud-optimized-geospatial-formats-guide/pr-preview/pr-121/
on branch gh-pages at 2024-11-15 00:08 UTC

[wildintellect]

@abarciauskas-bgse this is a great 1st version, a few questions and suggested fixes

• Fix: Compression - is currently a subheading under Consolidated Metadata
• Q: When talking about optimum chunk size is this compressed? Since compressed chunks should be delivered, I would think you want to target compressed sizes.
• Q: Additional Research, Chuck's example was on a non-cloud-opt HDF5, that's probably important.
• Fix: "How to check chunk size and shape" is missing output and explanation of how to read the output
• Q: Do we want to reference Zarr/Chunking in some way as alternatives, Zarr for when cloud native is fine and you don't need a single "archival file", Chucking (e.g. Kerchunk) when you want an index around and existing file you don't want to or can't change.

TODO: We'll open a different ticket for a notebook page about writing files from Python etc... rather than always having to repack existing files all the time.

[wildintellect]

A couple of fixes #121 (comment)

[betolink]

This looks good! just some minor additions that I'm not sure are relevant for a first pass on this.

• As of now, page size is across the board, this means that a user will have to find a balanced page size that reduces the metadata requests to the minimum at the same time that the unused size in the data chunks do not increase the file size by a lot... we noticed this for IS2 ATL03; example: file size 6GB, total metadata: ~20MB, if we use 8MB pages, the file size will increase by 1% approx, but for smaller files e.g. <1GB the 8MB page size increased the file size by ~10%. and this % varies depending on the page size vs data chunk size ratios. I short, a user should be careful to pick a page size as it's dataset dependent.
• The official HDF5 library needs to be configured to use the page aggregated files, I think @ajelenak said that this will change in March when the HDF Group releases the next major version.
• HDF5 doesn't have a geo spec for spatial chunking, at the lowest level if a user needs to subset data, the HDF5 library will have to load all the chunks of a dataset to create an index and use it to subset (e.g. lat lon subsetting), in order to take CO-HDF5 to the next level each chunk in the file should have a polygon/bbox info and it should be indexed in a way that the drivers can understand. This is related to over-reads, e.g. our data chunks are ~1MB per chunk, we use 8MB pages and in a subsetting operation we only need 2 chunks from contiguous pages... will be requesting 16MB instead of 2MB. @ajelenak can confirm if this is true. and @bilts also mentioned it on his ESIP talk.
• On creating vs repacking:
  • h5repack has some limitations on rechuking, the aggregation is fast but rechunking is really slow. Usually we want to increase chunk sizes so for ATL03 we had (Jeff Lee had to) use the h5py API directly to repack the files. This example can be used to see how to create CO-HDF5 with Python too: https://github.com/nsidc/cloud-optimized-icesat2/blob/main/notebooks/optimize-atl03.py
  • The netcdf library doesn't expose the low level hdf5 API so creating cloud optimized netcdf has to be done with h5repack or Python after the file is created. see: using the HDF5 file space strategy property Unidata/netcdf-c#2871 (reply in thread)

If I can think of more stuff I'll add it later (will be out next week). We also need to finish our tech report on IS2 and CO-HDF5, I think it should be ready for AGU.