If xarray supports the numpy.datetime64 data format and so does zarr, why is xarray converting my numpy.datetime64 time array to a float array when I write it to zarr?

[christine-e-smit]

This all started because I'm trying to understand how xarray handles fill values and time arrays of type numpy.datetime64 (a.k.a. M8[ns]). My fill values disappeared for some reason in the conversion to zarr and I discovered that xarray was converting my time array to floating point for some reason.

Here is a simple example:

import numpy as np
import numpy.ma as ma
import xarray as xr

# Create a two element time array where the first element is a fill value
time_array = np.array(['1999-12-31','2010-01-01'],
                      dtype="M8[ns]")
mask = np.array([True,False])
time_masked = ma.masked_array(time_array,mask,fill_value=np.datetime64('1999-12-31'))

xr_array = xr.DataArray(data=time_masked,dims=['time'],name="time")
xr_ds = xr.Dataset(dict(time=xr_array))
xr_ds.to_zarr("time_dataset.zarr")

This block of code produces the following warning:

/SOME_PATH/lib/python3.11/site-packages/xarray/coding/times.py:613: RuntimeWarning: invalid value encountered in cast
  int_num = np.asarray(num, dtype=np.int64)

If I then try to read the dataset back out again, I get more errors and the time fill values are set to 2010-01-01:

ds_read = xr.open_dataset("time_dataset.zarr")
print(ds_read["time"])

Read warnings:

/SOME_PATH/lib/python3.11/site-packages/xarray/backends/plugins.py:139: RuntimeWarning: 'netcdf4' fails while guessing
  warnings.warn(f"{engine!r} fails while guessing", RuntimeWarning)
/SOME_PATH/lib/python3.11/site-packages/xarray/coding/times.py:250: RuntimeWarning: invalid value encountered in cast
  flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(
/SOME_PATH/lib/python3.11/site-packages/xarray/coding/times.py:250: RuntimeWarning: invalid value encountered in cast
  flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(

Print out of time array, which no longer has a fill value in the first position:

<xarray.DataArray 'time' (time: 2)>
array(['2010-01-01T00:00:00.000000000', '2010-01-01T00:00:00.000000000'],
      dtype='datetime64[ns]')
Coordinates:
  * time     (time) datetime64[ns] 2010-01-01 2010-01-01

When I look in the actual files under time_dataset.zarr and look at the time_dataset.zarr/.zmetadata file created by xarray, the fill value has been changed to NaN and the datatype is <f8 rather than M8[ns]. It looks as though xarray recognized that these are dates and decided to encode them as NetCDF/CF-1 style "seconds since ". Apparently the fill value did not survive this translation.

{
    "metadata": {
        ".zattrs": {},
        ".zgroup": {
            "zarr_format": 2
        },
        "time/.zarray": {
            "chunks": [
                2
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                2
            ],
            "zarr_format": 2
        },
        "time/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "time"
            ],
            "calendar": "proleptic_gregorian",
            "units": "days since 2010-01-01 00:00:00"
        }
    },
    "zarr_consolidated_format": 1
}

I know xarray supports datetime64 values (https://docs.xarray.dev/en/stable/user-guide/time-series.html). I know zarr supports datetime64 values (https://zarr.readthedocs.io/en/stable/tutorial.html). So what is going on here?

[kmuehlbauer]

With latest xarray 2024.10.0 I get for xr_ds["time"] as well as ds_read["time"]:

<xarray.DataArray 'time' (time: 2)> Size: 16B
array([                          'NaT', '2010-01-01T00:00:00.000000000'],
      dtype='datetime64[ns]')
Coordinates:
  * time     (time) datetime64[ns] 16B NaT 2010-01-01

It seems that the issue was resolved over time.

INSTALLED VERSIONS
------------------
commit: None
python: 3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:23:07) [GCC 12.3.0]
python-bits: 64
OS: Linux
OS-release: 5.14.21-150500.55.68-default
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: ('de_DE', 'UTF-8')
libhdf5: 1.14.3
libnetcdf: 4.9.2

xarray: 2024.10.0
pandas: 2.2.3
numpy: 2.0.2
scipy: 1.14.0
netCDF4: 1.7.1
pydap: None
h5netcdf: 1.4.0
h5py: 3.11.0
zarr: 2.18.3
cftime: 1.6.4
nc_time_axis: None
iris: None
bottleneck: 1.4.0
dask: 2024.9.0
distributed: 2024.9.0
matplotlib: 3.9.2
cartopy: 0.24.0
seaborn: None
numbagg: None
fsspec: 2024.6.1
cupy: None
pint: 0.24.3
sparse: None
flox: None
numpy_groupies: None
setuptools: 70.1.1
pip: 24.0
conda: None
pytest: 8.2.2
mypy: None
IPython: 8.25.0
sphinx: 8.1.0