Request for Standardized FixedScaleOffset Codec Extension

[emmanuelmathot]

Problem Statement

Float-to-integer quantization with scale/offset transformation is a common pattern for reducing storage size while maintaining precision in Earth observation and climate data. However, there is currently no stable, standardized Zarr codec for this operation, leading to ecosystem fragmentation and interoperability issues.

Current Fragmented State

Multiple incompatible implementations exist:

• Zarr V2/V3: scale_factor and add_offset attributes (inherited from CF/netCDF conventions, handled implicitly by xarray)
• Zarr V3: numcodecs.fixedscaleoffset codec (implemented in Python numcodecs, but not consistently available in JavaScript libraries like zarrita.js or numcodecs.js)

Real-World Impact

This fragmentation prevents data from being accessible across the ecosystem. During development of the EOPF Sentinel Zarr Explorer for ESA, we discovered that Sentinel-2 data encoded with scale/offset at native resolutions (10m, 20m, 60m) cannot be visualized in web contexts using OpenLayers because the codec is unavailable in JavaScript implementations.

Request

The Zarr community should converge on a stable, registered extension for fixedscaleoffset transformation that:

1. Works consistently across Zarr V2 and V3 and is implementation independent
2. Has reference implementations in both Python and JavaScript
3. Defines clear semantics for the transformation parameters (offset, scale, dtype, astype). codec? filter?
4. Follows the extension registration process outlined in this repository

Related Discussions

• numcodecs.js issue: Support for fixedscaleoffset manzt/numcodecs.js#49
• zarrita.js PR (stalled): Implement fixedscaleoffset codec manzt/zarrita.js#312
• EOPF data-model conversion issue: FixedScaleOffset codec not preserved during zarr v2 to v3 conversion EOPF-Explorer/data-model#106

cc @manzt @d-v-b @ahocevar @vincentsarago @vdumoul @maxrjones @abarciauskas-bgse @j08lue

[d-v-b]

see also https://github.com/zarr-developers/zarr-extensions/pull/2

IMO this will be very straightforward to standardize. we should drop the numcodecs prefix from the name, and settle a few details about the semantics of data type identifiers. @normanrz are you OK with breaking this out into a separate PR? I'm happy to work on it.