You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Related Issue:
Intake_esm catalog hosted on an s3 file system which points to kerchunk files also hosted on the same file system cannot be accessed properly by existing intake_esm scripts.
Solution:
A small change to line 53 onwards in intake_esm/source.py:
if data_format == 'reference':
if 's3://' in urlpath:
import s3fs
xarray_open_kwargs['backend_kwargs']['storage_options']['fo'] = s3fs.S3FileSystem(**kwargs).open(urlpath, **kwargs)
else:
xarray_open_kwargs['backend_kwargs']['storage_options']['fo'] = urlpath
Alternatives:
There might be other options to consider/features in intake_esm I'm not taking advantage of, if anyone has any insight let me know.
The above change relates to a later section: fsspec.get_mapper('reference://', fo=ref ...)
Where for a kerchunk file on s3, ref needs to be the S3FileSystem object not the string path.
The text was updated successfully, but these errors were encountered:
Hi @andersy005, hope you are well.
Does this fix look okay to you? It would help us with providing access to Kerchunk files over S3 (with authz). Thanks
Related Issue:
Intake_esm catalog hosted on an s3 file system which points to kerchunk files also hosted on the same file system cannot be accessed properly by existing intake_esm scripts.
Solution:
A small change to line 53 onwards in intake_esm/source.py:
Alternatives:
There might be other options to consider/features in intake_esm I'm not taking advantage of, if anyone has any insight let me know.
The above change relates to a later section:
fsspec.get_mapper('reference://', fo=ref ...)
Where for a kerchunk file on s3, ref needs to be the S3FileSystem object not the string path.
The text was updated successfully, but these errors were encountered: