ZI drivers + Rarr implementation

[joshmoore]

In an attempt to:

• speed up tests
• make them more manageable
• ease local testing
• encourage the creation of more tests
• clearly identify what each implementation supports
• etc.

This PR:

• moves each implementation to its own subdirectory
• creates a conda environment for each
• runs each in a separate GHA job
• generates an output page for each
  • or alternatively: Python support dorny/test-reporter#244
• ... with timings

[joshmoore]

Various updates. make list now prints this:

../../data/js.zr	gzip
../../data/js.zr	blosc/lz4
../../data/js.zr	zlib
../../data/js.zr	raw
../../data/jzarr_flat.zr	raw
../../data/jzarr_flat.zr	zlib
../../data/jzarr_flat.zr	blosc/lz4
../../data/n5-java.n5	raw
../../data/n5-java.n5	bzip2
../../data/n5-java.n5	gzip
../../data/n5-java.n5	lz4
../../data/n5-java.n5	xz
../../data/pyn5.n5	raw
../../data/pyn5.n5	bzip2
../../data/pyn5.n5	gzip
../../data/z5py.zr	gzip
../../data/z5py.zr	blosc/lz4
../../data/z5py.zr	zlib
../../data/z5py.zr	raw
../../data/z5py.n5	gzip
../../data/z5py.n5	raw
../../data/zarr_DirectoryStore_flat.zr	gzip
../../data/zarr_DirectoryStore_flat.zr	blosc/lz4
../../data/zarr_DirectoryStore_flat.zr	zlib
../../data/zarr_DirectoryStore_flat.zr	raw
../../data/zarr_FSStore_flat.zr	gzip
../../data/zarr_FSStore_flat.zr	blosc/lz4
../../data/zarr_FSStore_flat.zr	zlib
../../data/zarr_FSStore_flat.zr	raw
../../data/zarr_NestedDirectoryStore_nested.zr	gzip
../../data/zarr_NestedDirectoryStore_nested.zr	blosc/lz4
../../data/zarr_NestedDirectoryStore_nested.zr	zlib
../../data/zarr_NestedDirectoryStore_nested.zr	raw
../../data/zarr_FSStore_nested.zr	gzip
../../data/zarr_FSStore_nested.zr	blosc/lz4
../../data/zarr_FSStore_nested.zr	zlib
../../data/zarr_FSStore_nested.zr	raw
../../data/zarr.n5	gzip
../../data/zarr.n5	raw
../../data/zarrita	gzip
../../data/zarrita	blosc/lz4
../../data/zarrita	raw
../../data/zarrita_sharded	gzip
../../data/zarrita_sharded	blosc/lz4
../../data/zarrita_sharded	raw
../../data/zarrita_nested	gzip
../../data/zarrita_nested	blosc/lz4
../../data/zarrita_nested	raw
../../data/zarrita_nested_sharded	gzip
../../data/zarrita_nested_sharded	blosc/lz4
../../data/zarrita_nested_sharded	raw

[joshmoore]

Locally I'm now seeing this:

$ pytest test/test_read_all.py
====================================================================================
platform darwin -- Python 3.12.4, pytest-8.3.1, pluggy-1.5.0
rootdir: /Users/jamoore/opt/zarr_implementations
collected 41 items

test/test_read_all.py ...ssss....ssssssss...sss...sssssssssss..                                                                                                                                                        [100%]

====================================================================================

I will likely need to add a few .skip files (e.g., for pyn5) but the current failures on GitHub actions are across the board:

import-im6.q16: unable to open X server `' @ error/import.c/ImportImageCommand/346.

Still trying to track that down if anyone has seen it before.

[joshmoore]

============================= test session starts ==============================
platform linux -- Python 3.12.4, pytest-8.3.1, pluggy-1.5.0
rootdir: /home/runner/work/zarr_implementations/zarr_implementations
collected 123 items

test/test_read_all.py ...ssss....ssssssss...sss...ssssss...ssss....sssss [ 40%]
sss...ssss....ssssssss...ssss....ssssssss...ssss....sssssssssssss..ssss. [ 99%]
.                                                                        [100%]

================== 45 passed, 78 skipped in 113.97s (0:01:53) ==================

I'd propose getting this merged and then start adding in other implementations including converters like https://github.com/glencoesoftware/zarr2zarr and https://github.com/LDeakin/zarrs_tools

cc: @MSanKeys963 @sbesson

[sanketverma1704]

I'd propose getting this merged and then start adding in other implementations including converters like https://github.com/glencoesoftware/zarr2zarr and https://github.com/LDeakin/zarrs_tools

Would having a page like https://zarr.dev/tools and listing the converters make sense?

[joshmoore]

Would having a page like https://zarr.dev/tools and listing the converters make sense?

Eventually. Let's get some more of the testing done first though.

[joshmoore]

cc: @perlman

[joshmoore]

Discussions during Zarr Summit

• Target .github/workflow:

... build and install here ...
gh repo clone zarr-developers/zarr-conformance-datasets   # ds001, ds002, ...
gh repo clone zarr-developers/zarr-conformance-tests      # t0001, t0002, ....
export conformance_datasets=$PWD/zarr-conformance-datasets
export conformance_command="rarr compat" or "docker run -ti --rm my_image"
zarr-conformance-tests/run.sh
# Or: git clone https://github.com/aiiie/cram.git
#     . zarr-conformance-tests/setup.sh
#     cram

(These could also be GitHub actions.)

• Target test file:

  $ $conformance_command slide_1d $conformance_datasets/ds001 \
      --array_name=foo \
      --dim=1 --start=4 --stop=6 --step=2
  0 1 2 3 4

• where the current extent of this repo is essentially:

  $ $conformance_command generate data.zarr
  $ $conformance_command verify data.zarr --codec=lz4

• Additional functionality could include piping:

  $ cat raw_bytes.dat | $conformance_command write local0001.zarr --shape=64,64,3 --dtype=int8 --array_name=foo --codec=lz4
  $ $conformance_command slide_1d local0001.zarr \
      --array_name=foo \
      --dim=1 --start=4 --stop=6 --step=2
  0 1 2 3 4

• Other write commands/tests might:
  • update a range of a slice (e.g. set_1d)
  • generate large tests (e.g. for multi-part uploads), perhaps for a fill_range command

TODOs:

• Merged this PR and iterate from there
• Move from Conda to Docker
• Move to existing tool (like cram)
• Simple Drop-in tests
• Decentralizing
  • Extract commands to the individual repos
  • Add .zarr_ci callback (see infra)
• Opt-in subsets for automated
• Standard subsets (Required, Recommended, Optional)
• Define own subsets (e.g. Rarr has no write)
• Central repo (which also uses a regression subset, "previously passed should still pass")
• Code coverage during the test suite

cc: @clbarnes

[Bisaloo]

I have a very initial implementation here: https://github.com/Bisaloo/zarr-conformance-tests.

I'll polish it tomorrow and post it in the zulip for feedback.