Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit test failure on Debian Sid #3001

Open
avalentino opened this issue Nov 23, 2024 · 9 comments
Open

Unit test failure on Debian Sid #3001

avalentino opened this issue Nov 23, 2024 · 9 comments

Comments

@avalentino
Copy link
Contributor

Describe the bug
Some of the unit tests fail on Debian Sid (x86_64)

To Reproduce

$ python3 -m pytest satpy/tests

Expected behavior
All tests pass

Actual results
Python 3.12

=================================== FAILURES ===================================
_____ TestSunZenithCorrector.test_basic_default_provided[sunz_ds1-float32] _____

self = <satpy.tests.test_modifiers.TestSunZenithCorrector object at 0x759c0ff72930>
data_arr = <xarray.DataArray 'ones_like-094cd22f1fd7da5c912d599befe75b96' (y: 2, x: 2)> Size: 32B
dask.array<ones_like, shape=(2,... test\nProjection ID: test\nProje...
    start_time:  2018-01-01 18:00:00
    modifiers:   ()
    name:        test_vis
sunz_sza = <xarray.DataArray 'rad2deg-224875e7aff21971b18fc6dd0687aa42' (y: 2, x: 2)> Size: 32B
dask.array<rad2deg, shape=(2, 2),...x        (x) int64 16B 0 1
Attributes:
    area:     Area ID: test\nDescription: test\nProjection ID: test\nProjecti...
dtype = <class 'numpy.float32'>

    @pytest.mark.parametrize("dtype", [np.float32, np.float64])
    @pytest.mark.parametrize("data_arr", [lazy_fixture("sunz_ds1"), lazy_fixture("sunz_ds1_stacked")])
    def test_basic_default_provided(self, data_arr, sunz_sza, dtype):
        """Test default limits when SZA is provided."""
        from satpy.modifiers.geometry import SunZenithCorrector
        comp = SunZenithCorrector(name="sza_test", modifiers=tuple())
        res = comp((data_arr.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
        expected = np.array([[22.401667, 22.31777], [22.437503, 22.353533]], dtype=dtype)
        values = res.values
>       np.testing.assert_allclose(values, expected)

satpy/tests/test_modifiers.py:162:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<function assert_allclose.<locals>.compare at 0x759bb02d7420>, array([[22.401634, 22.317774],
       [22.437466, 22.353533]], dtype=float32), array([[22.401667, 22.31777 ],
       [22.437504, 22.353533]], dtype=float32))
kwds = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-07, atol=0', 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError:
E           Not equal to tolerance rtol=1e-07, atol=0
E
E           Mismatched elements: 3 / 4 (75%)
E           Max absolute difference: 3.8146973e-05
E           Max relative difference: 1.7001433e-06
E            x: array([[22.401634, 22.317774],
E                  [22.437466, 22.353533]], dtype=float32)
E            y: array([[22.401667, 22.31777 ],
E                  [22.437504, 22.353533]], dtype=float32)

/usr/lib/python3.12/contextlib.py:81: AssertionError
_ TestSunZenithCorrector.test_basic_default_provided[sunz_ds1_stacked-float32] _

self = <satpy.tests.test_modifiers.TestSunZenithCorrector object at 0x759c0ff702c0>
data_arr = <xarray.DataArray 'ones_like-094cd22f1fd7da5c912d599befe75b96' (y: 2, x: 2)> Size: 32B
dask.array<ones_like, shape=(2,...dAreaDefinition object at 0x759c1...
    start_time:  2018-01-01 18:00:00
    modifiers:   ()
    name:        test_vis
sunz_sza = <xarray.DataArray 'rad2deg-224875e7aff21971b18fc6dd0687aa42' (y: 2, x: 2)> Size: 32B
dask.array<rad2deg, shape=(2, 2),...x        (x) int64 16B 0 1
Attributes:
    area:     Area ID: test\nDescription: test\nProjection ID: test\nProjecti...
dtype = <class 'numpy.float32'>

    @pytest.mark.parametrize("dtype", [np.float32, np.float64])
    @pytest.mark.parametrize("data_arr", [lazy_fixture("sunz_ds1"), lazy_fixture("sunz_ds1_stacked")])
    def test_basic_default_provided(self, data_arr, sunz_sza, dtype):
        """Test default limits when SZA is provided."""
        from satpy.modifiers.geometry import SunZenithCorrector
        comp = SunZenithCorrector(name="sza_test", modifiers=tuple())
        res = comp((data_arr.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
        expected = np.array([[22.401667, 22.31777], [22.437503, 22.353533]], dtype=dtype)
        values = res.values
>       np.testing.assert_allclose(values, expected)

satpy/tests/test_modifiers.py:162:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<function assert_allclose.<locals>.compare at 0x759bb02d59e0>, array([[22.401634, 22.317774],
       [22.437466, 22.353533]], dtype=float32), array([[22.401667, 22.31777 ],
       [22.437504, 22.353533]], dtype=float32))
kwds = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-07, atol=0', 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError:
E           Not equal to tolerance rtol=1e-07, atol=0
E
E           Mismatched elements: 3 / 4 (75%)
E           Max absolute difference: 3.8146973e-05
E           Max relative difference: 1.7001433e-06
E            x: array([[22.401634, 22.317774],
E                  [22.437466, 22.353533]], dtype=float32)
E            y: array([[22.401667, 22.31777 ],
E                  [22.437504, 22.353533]], dtype=float32)

/usr/lib/python3.12/contextlib.py:81: AssertionError
=============================== warnings summary ===============================

[...]

=========================== short test summary info ============================
FAILED satpy/tests/test_modifiers.py::TestSunZenithCorrector::test_basic_default_provided[sunz_ds1-float32]
FAILED satpy/tests/test_modifiers.py::TestSunZenithCorrector::test_basic_default_provided[sunz_ds1_stacked-float32]

Python 3.13


=================================== FAILURES ===================================
_____ TestSunZenithCorrector.test_basic_default_provided[sunz_ds1-float32] _____

self = <satpy.tests.test_modifiers.TestSunZenithCorrector object at 0x7d9dc44b1490>
data_arr = <xarray.DataArray 'ones_like-c125f88844151bd873ede2ef3b9aab56' (y: 2, x: 2)> Size: 32B
dask.array<ones_like, shape=(2,... test\nProjection ID: test\nProje...
    start_time:  2018-01-01 18:00:00
    modifiers:   ()
    name:        test_vis
sunz_sza = <xarray.DataArray 'rad2deg-224875e7aff21971b18fc6dd0687aa42' (y: 2, x: 2)> Size: 32B
dask.array<rad2deg, shape=(2, 2),...x        (x) int64 16B 0 1
Attributes:
    area:     Area ID: test\nDescription: test\nProjection ID: test\nProjecti...
dtype = <class 'numpy.float32'>

    @pytest.mark.parametrize("dtype", [np.float32, np.float64])
    @pytest.mark.parametrize("data_arr", [lazy_fixture("sunz_ds1"), lazy_fixture("sunz_ds1_stacked")])
    def test_basic_default_provided(self, data_arr, sunz_sza, dtype):
        """Test default limits when SZA is provided."""
        from satpy.modifiers.geometry import SunZenithCorrector
        comp = SunZenithCorrector(name="sza_test", modifiers=tuple())
        res = comp((data_arr.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
        expected = np.array([[22.401667, 22.31777], [22.437503, 22.353533]], dtype=dtype)
        values = res.values
>       np.testing.assert_allclose(values, expected)

satpy/tests/test_modifiers.py:162:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<function assert_allclose.<locals>.compare at 0x7d9d58d23920>, array([[22.401634, 22.317774],
       [22.437466, 22.353533]], dtype=float32), array([[22.401667, 22.31777 ],
       [22.437504, 22.353533]], dtype=float32))
kwds = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-07, atol=0', 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError:
E           Not equal to tolerance rtol=1e-07, atol=0
E
E           Mismatched elements: 3 / 4 (75%)
E           Max absolute difference: 3.8146973e-05
E           Max relative difference: 1.7001433e-06
E            x: array([[22.401634, 22.317774],
E                  [22.437466, 22.353533]], dtype=float32)
E            y: array([[22.401667, 22.31777 ],
E                  [22.437504, 22.353533]], dtype=float32)

/usr/lib/python3.13/contextlib.py:85: AssertionError
_ TestSunZenithCorrector.test_basic_default_provided[sunz_ds1_stacked-float32] _

self = <satpy.tests.test_modifiers.TestSunZenithCorrector object at 0x7d9dc3b0f240>
data_arr = <xarray.DataArray 'ones_like-c125f88844151bd873ede2ef3b9aab56' (y: 2, x: 2)> Size: 32B
dask.array<ones_like, shape=(2,...dAreaDefinition object at 0x7d9d5...
    start_time:  2018-01-01 18:00:00
    modifiers:   ()
    name:        test_vis
sunz_sza = <xarray.DataArray 'rad2deg-224875e7aff21971b18fc6dd0687aa42' (y: 2, x: 2)> Size: 32B
dask.array<rad2deg, shape=(2, 2),...x        (x) int64 16B 0 1
Attributes:
    area:     Area ID: test\nDescription: test\nProjection ID: test\nProjecti...
dtype = <class 'numpy.float32'>

    @pytest.mark.parametrize("dtype", [np.float32, np.float64])
    @pytest.mark.parametrize("data_arr", [lazy_fixture("sunz_ds1"), lazy_fixture("sunz_ds1_stacked")])
    def test_basic_default_provided(self, data_arr, sunz_sza, dtype):
        """Test default limits when SZA is provided."""
        from satpy.modifiers.geometry import SunZenithCorrector
        comp = SunZenithCorrector(name="sza_test", modifiers=tuple())
        res = comp((data_arr.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
        expected = np.array([[22.401667, 22.31777], [22.437503, 22.353533]], dtype=dtype)
        values = res.values
>       np.testing.assert_allclose(values, expected)

satpy/tests/test_modifiers.py:162:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<function assert_allclose.<locals>.compare at 0x7d9d5ab26200>, array([[22.401634, 22.317774],
       [22.437466, 22.353533]], dtype=float32), array([[22.401667, 22.31777 ],
       [22.437504, 22.353533]], dtype=float32))
kwds = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-07, atol=0', 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError:
E           Not equal to tolerance rtol=1e-07, atol=0
E
E           Mismatched elements: 3 / 4 (75%)
E           Max absolute difference: 3.8146973e-05
E           Max relative difference: 1.7001433e-06
E            x: array([[22.401634, 22.317774],
E                  [22.437466, 22.353533]], dtype=float32)
E            y: array([[22.401667, 22.31777 ],
E                  [22.437504, 22.353533]], dtype=float32)

/usr/lib/python3.13/contextlib.py:85: AssertionError
_________________ TestFSFile.test_open_zip_fs_regular_filename _________________

self = <satpy.tests.test_readers.TestFSFile object at 0x7d9dc3b32050>
local_filename2 = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335')
local_zip_file = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335.zip')

    def test_open_zip_fs_regular_filename(self, local_filename2, local_zip_file):
        """Test opening a zipfs with a regular filename provided."""
        from fsspec.implementations.zip import ZipFileSystem

        from satpy.readers import FSFile
>       zip_fs = ZipFileSystem(local_zip_file)

satpy/tests/test_readers.py:1081:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3/dist-packages/fsspec/spec.py:81: in __call__
    obj = super().__call__(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Archive-like object ZipFileSystem at 138116555610448>
fo = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335.zip')
mode = 'r', target_protocol = None, target_options = None, compression = 0
allowZip64 = True, compresslevel = None, kwargs = {}

    def __init__(
        self,
        fo="",
        mode="r",
        target_protocol=None,
        target_options=None,
        compression=zipfile.ZIP_STORED,
        allowZip64=True,
        compresslevel=None,
        **kwargs,
    ):
        """
        Parameters
        ----------
        fo: str or file-like
            Contains ZIP, and must exist. If a str, will fetch file using
            :meth:`~fsspec.open_files`, which must return one file exactly.
        mode: str
            Accept: "r", "w", "a"
        target_protocol: str (optional)
            If ``fo`` is a string, this value can be used to override the
            FS protocol inferred from a URL
        target_options: dict (optional)
            Kwargs passed when instantiating the target FS, if ``fo`` is
            a string.
        compression, allowZip64, compresslevel: passed to ZipFile
            Only relevant when creating a ZIP
        """
        super().__init__(self, **kwargs)
        if mode not in set("rwa"):
            raise ValueError(f"mode '{mode}' no understood")
        self.mode = mode
        if isinstance(fo, str):
            if mode == "a":
                m = "r+b"
            else:
                m = mode + "b"
            fo = fsspec.open(
                fo, mode=m, protocol=target_protocol, **(target_options or {})
            )
        self.force_zip_64 = allowZip64
        self.of = fo
>       self.fo = fo.__enter__()  # the whole instance is a context
E       AttributeError: 'PosixPath' object has no attribute '__enter__'. Did you mean: '__bytes__'?

/usr/lib/python3/dist-packages/fsspec/implementations/zip.py:61: AttributeError
_______________________ TestFSFile.test_sorting_fsfiles ________________________

self = <satpy.tests.test_readers.TestFSFile object at 0x7d9dc3b13020>
local_filename = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files0/fd3abda6-a9be-11ef-a36d-974f9343c335')
local_filename2 = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335')
local_zip_file = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335.zip')

    def test_sorting_fsfiles(self, local_filename, local_filename2, local_zip_file):
        """Test sorting FSFiles."""
        from fsspec.implementations.zip import ZipFileSystem

        from satpy.readers import FSFile
>       zip_fs = ZipFileSystem(local_zip_file)

satpy/tests/test_readers.py:1099:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3/dist-packages/fsspec/spec.py:81: in __call__
    obj = super().__call__(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Archive-like object ZipFileSystem at 138114943746128>
fo = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335.zip')
mode = 'r', target_protocol = None, target_options = None, compression = 0
allowZip64 = True, compresslevel = None, kwargs = {}

    def __init__(
        self,
        fo="",
        mode="r",
        target_protocol=None,
        target_options=None,
        compression=zipfile.ZIP_STORED,
        allowZip64=True,
        compresslevel=None,
        **kwargs,
    ):
        """
        Parameters
        ----------
        fo: str or file-like
            Contains ZIP, and must exist. If a str, will fetch file using
            :meth:`~fsspec.open_files`, which must return one file exactly.
        mode: str
            Accept: "r", "w", "a"
        target_protocol: str (optional)
            If ``fo`` is a string, this value can be used to override the
            FS protocol inferred from a URL
        target_options: dict (optional)
            Kwargs passed when instantiating the target FS, if ``fo`` is
            a string.
        compression, allowZip64, compresslevel: passed to ZipFile
            Only relevant when creating a ZIP
        """
        super().__init__(self, **kwargs)
        if mode not in set("rwa"):
            raise ValueError(f"mode '{mode}' no understood")
        self.mode = mode
        if isinstance(fo, str):
            if mode == "a":
                m = "r+b"
            else:
                m = mode + "b"
            fo = fsspec.open(
                fo, mode=m, protocol=target_protocol, **(target_options or {})
            )
        self.force_zip_64 = allowZip64
        self.of = fo
>       self.fo = fo.__enter__()  # the whole instance is a context
E       AttributeError: 'PosixPath' object has no attribute '__enter__'. Did you mean: '__bytes__'?

/usr/lib/python3/dist-packages/fsspec/implementations/zip.py:61: AttributeError
___________________________ TestFSFile.test_equality ___________________________

self = <satpy.tests.test_readers.TestFSFile object at 0x7d9dc4312510>
local_filename = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files0/fd3abda6-a9be-11ef-a36d-974f9343c335')
local_filename2 = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335')
local_zip_file = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335.zip')

    def test_equality(self, local_filename, local_filename2, local_zip_file):
        """Test that FSFile compares equal when it should."""
        from fsspec.implementations.zip import ZipFileSystem

        from satpy.readers import FSFile
>       zip_fs = ZipFileSystem(local_zip_file)

satpy/tests/test_readers.py:1114:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3/dist-packages/fsspec/spec.py:81: in __call__
    obj = super().__call__(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Archive-like object ZipFileSystem at 138114943746736>
fo = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335.zip')
mode = 'r', target_protocol = None, target_options = None, compression = 0
allowZip64 = True, compresslevel = None, kwargs = {}

    def __init__(
        self,
        fo="",
        mode="r",
        target_protocol=None,
        target_options=None,
        compression=zipfile.ZIP_STORED,
        allowZip64=True,
        compresslevel=None,
        **kwargs,
    ):
        """
        Parameters
        ----------
        fo: str or file-like
            Contains ZIP, and must exist. If a str, will fetch file using
            :meth:`~fsspec.open_files`, which must return one file exactly.
        mode: str
            Accept: "r", "w", "a"
        target_protocol: str (optional)
            If ``fo`` is a string, this value can be used to override the
            FS protocol inferred from a URL
        target_options: dict (optional)
            Kwargs passed when instantiating the target FS, if ``fo`` is
            a string.
        compression, allowZip64, compresslevel: passed to ZipFile
            Only relevant when creating a ZIP
        """
        super().__init__(self, **kwargs)
        if mode not in set("rwa"):
            raise ValueError(f"mode '{mode}' no understood")
        self.mode = mode
        if isinstance(fo, str):
            if mode == "a":
                m = "r+b"
            else:
                m = mode + "b"
            fo = fsspec.open(
                fo, mode=m, protocol=target_protocol, **(target_options or {})
            )
        self.force_zip_64 = allowZip64
        self.of = fo
>       self.fo = fo.__enter__()  # the whole instance is a context
E       AttributeError: 'PosixPath' object has no attribute '__enter__'. Did you mean: '__bytes__'?

/usr/lib/python3/dist-packages/fsspec/implementations/zip.py:61: AttributeError
_____________________________ TestFSFile.test_hash _____________________________

self = <satpy.tests.test_readers.TestFSFile object at 0x7d9dc43127b0>
local_filename = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files0/fd3abda6-a9be-11ef-a36d-974f9343c335')
local_filename2 = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335')
local_zip_file = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335.zip')

    def test_hash(self, local_filename, local_filename2, local_zip_file):
        """Test that FSFile hashing behaves sanely."""
        from fsspec.implementations.cached import CachingFileSystem
        from fsspec.implementations.local import LocalFileSystem
        from fsspec.implementations.zip import ZipFileSystem

        from satpy.readers import FSFile

        lfs = LocalFileSystem()
>       zfs = ZipFileSystem(local_zip_file)

satpy/tests/test_readers.py:1129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3/dist-packages/fsspec/spec.py:81: in __call__
    obj = super().__call__(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Archive-like object ZipFileSystem at 138115157020976>
fo = PosixPath('/tmp/pytest-of-pbuilder/pytest-0/local_files1/fd4a8254-a9be-11ef-a36d-974f9343c335.zip')
mode = 'r', target_protocol = None, target_options = None, compression = 0
allowZip64 = True, compresslevel = None, kwargs = {}

    def __init__(
        self,
        fo="",
        mode="r",
        target_protocol=None,
        target_options=None,
        compression=zipfile.ZIP_STORED,
        allowZip64=True,
        compresslevel=None,
        **kwargs,
    ):
        """
        Parameters
        ----------
        fo: str or file-like
            Contains ZIP, and must exist. If a str, will fetch file using
            :meth:`~fsspec.open_files`, which must return one file exactly.
        mode: str
            Accept: "r", "w", "a"
        target_protocol: str (optional)
            If ``fo`` is a string, this value can be used to override the
            FS protocol inferred from a URL
        target_options: dict (optional)
            Kwargs passed when instantiating the target FS, if ``fo`` is
            a string.
        compression, allowZip64, compresslevel: passed to ZipFile
            Only relevant when creating a ZIP
        """
        super().__init__(self, **kwargs)
        if mode not in set("rwa"):
            raise ValueError(f"mode '{mode}' no understood")
        self.mode = mode
        if isinstance(fo, str):
            if mode == "a":
                m = "r+b"
            else:
                m = mode + "b"
            fo = fsspec.open(
                fo, mode=m, protocol=target_protocol, **(target_options or {})
            )
        self.force_zip_64 = allowZip64
        self.of = fo
>       self.fo = fo.__enter__()  # the whole instance is a context
E       AttributeError: 'PosixPath' object has no attribute '__enter__'. Did you mean: '__bytes__'?

/usr/lib/python3/dist-packages/fsspec/implementations/zip.py:61: AttributeError
=============================== warnings summary ===============================

[...]

=========================== short test summary info ============================
FAILED satpy/tests/test_modifiers.py::TestSunZenithCorrector::test_basic_default_provided[sunz_ds1-float32]
FAILED satpy/tests/test_modifiers.py::TestSunZenithCorrector::test_basic_default_provided[sunz_ds1_stacked-float32]
FAILED satpy/tests/test_readers.py::TestFSFile::test_open_zip_fs_regular_filename
FAILED satpy/tests/test_readers.py::TestFSFile::test_sorting_fsfiles - Attrib...
FAILED satpy/tests/test_readers.py::TestFSFile::test_equality - AttributeErro...
FAILED satpy/tests/test_readers.py::TestFSFile::test_hash - AttributeError: '...
= 6 failed, 2472 passed, 2 skipped, 263 deselected, 6 xfailed, 584 warnings in 834.57s (0:13:54) =

Environment Info:

  • OS: Linux
  • Satpy Version: 0.53.0
  • PyResample Version: 1.31
  • Readers and writers dependencies (when relevant):
$ python3 -c "from satpy.utils import check_satpy; check_satpy()"
Readers
=======
/usr/lib/python3/dist-packages/dask/dataframe/__init__.py:31: FutureWarning: 
Dask dataframe query planning is disabled because dask-expr is not installed.

You can install it with `pip install dask[dataframe]` or `conda install dask`.
This will raise in a future version.

 warnings.warn(msg, FutureWarning)
abi_l1b:  ok
abi_l1b_scmi:  ok
abi_l2_nc:  ok
acspo:  ok
agri_fy4a_l1:  ok
agri_fy4b_l1:  ok
ahi_hrit:  ok
ahi_hsd:  ok
ahi_l1b_gridded_bin:  ok
ahi_l2_nc:  ok
ami_l1b:  ok
amsr2_l1b:  ok
amsr2_l2:  ok
amsr2_l2_gaasp:  ok
amsub_l1c_aapp:  ok
ascat_l2_soilmoisture_bufr:  ok
atms_l1b_nc:  ok
atms_sdr_hdf5:  ok
avhrr_l1b_aapp:  ok
avhrr_l1b_eps:  ok
avhrr_l1b_gaclac:  ok
avhrr_l1b_hrpt:  ok
avhrr_l1c_eum_gac_fdr_nc:  ok
caliop_l2_cloud:  cannot find module 'satpy.readers.caliop_l2_cloud' (cannot import name 'Dataset' from 'satpy.dataset' (/home/antonio/debian/git/satpy/satpy/dataset/__init__.py))
clavrx:  ok
cmsaf-claas2_l2_nc:  ok
electrol_hrit:  ok
epic_l1b_h5:  ok
fci_l1c_nc:  ok
fci_l2_bufr:  ok
fci_l2_grib:  ok
fci_l2_nc:  ok
fy3a_mersi1_l1b:  ok
fy3b_mersi1_l1b:  ok
fy3c_mersi1_l1b:  ok
generic_image:  ok
geocat:  ok
gerb_l2_hr_h5:  ok
ghi_l1:  ok
ghrsst_l2:  ok
gld360_ualf2:  ok
glm_l2:  ok
gms5-vissr_l1b:  ok
goci2_l2_nc:  ok
goes-imager_hrit:  ok
goes-imager_nc:  ok
gpm_imerg:  ok
grib:  ok
hsaf_grib:  ok
hsaf_h5:  ok
hy2_scat_l2b_h5:  ok
iasi_l2:  ok
iasi_l2_cdr_nc:  ok
iasi_l2_so2_bufr:  ok
ici_l1b_nc:  ok
insat3d_img_l1b_h5:  ok
jami_hrit:  ok
li_l2_nc:  ok
maia:  ok
mcd12q1:  ok
meris_nc_sen3:  ok
mersi2_l1b:  ok
mersi3_l1b:  ok
mersi_ll_l1b:  ok
mersi_rm_l1b:  ok
mhs_l1c_aapp:  ok
mimicTPW2_comp:  ok
mirs:  ok
modis_l1b:  ok
modis_l2:  ok
modis_l3:  ok
msi_safe:  ok
msi_safe_l2a:  ok
msu_gsa_l1b:  ok
mtsat2-imager_hrit:  ok
mviri_l1b_fiduceo_nc:  ok
mwi_l1b_nc:  ok
mws_l1b_nc:  ok
nucaps:  ok
nwcsaf-geo:  ok
nwcsaf-msg2013-hdf5:  ok
nwcsaf-pps_nc:  ok
oceancolorcci_l3_nc:  ok
oci_l2_bgc:  ok
olci_l1b:  ok
olci_l2:  ok
oli_tirs_l1_tif:  ok
omps_edr:  ok
osisaf_nc:  ok
safe_sar_l2_ocn:  ok
sar-c_safe:  ok
satpy_cf_nc:  ok
scatsat1_l2b:  cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/home/antonio/debian/git/satpy/satpy/dataset/__init__.py))
seadas_l2:  ok
seviri_l1b_hrit:  ok
seviri_l1b_icare:  ok
seviri_l1b_native:  ok
seviri_l1b_nc:  ok
seviri_l2_bufr:  ok
seviri_l2_grib:  ok
sgli_l1b:  ok
slstr_l1b:  ok
smos_l2_wind:  ok
tropomi_l2:  ok
vii_l1b_nc:  ok
vii_l2_nc:  ok
viirs_compact:  ok
viirs_edr:  ok
viirs_edr_active_fires:  ok
viirs_edr_flood:  ok
viirs_l1b:  ok
viirs_l2:  ok
viirs_sdr:  ok
viirs_vgac_l1c_nc:  ok
virr_l1b:  ok

Writers
=======
/usr/lib/python3/dist-packages/pyninjotiff/tifffile.py:154: UserWarning: failed to import the optional _tifffile C extension module.
Loading of some compressed images will be slow.
Tifffile.c can be obtained at http://www.lfd.uci.edu/~gohlke/
 warnings.warn(
awips_tiled:  ok
cf:  ok
geotiff:  ok
mitiff:  ok
ninjogeotiff:  ok
ninjotiff:  ok
simple_image:  ok

Versions
======
platform: Linux-6.11.0-9-generic-x86_64-with-glibc2.40
python: 3.12.7

cartopy: 0.24.1
dask: 2024.5.2+dfsg
fsspec: 2024.9.0
gdal: 3.9.3
geoviews: not installed
h5netcdf: 1.4.0
h5py: 3.11.0
netcdf4: 1.7.2
numpy: 1.26.4
pyhdf: 0.11.4
pyproj: 3.7.0
rasterio: 1.4.2
xarray: 2024.9.0

Additional context
Add any other context about the problem here.

@djhoese
Copy link
Member

djhoese commented Nov 27, 2024

Do you know what other packages were updated since the last time these tests were run on debian?

@avalentino
Copy link
Contributor Author

Sorry I cannot say for sure what packages have been updated since the last build.

Most probably

  • scipy 1.13.1 --> 1.14.1
  • xarray 2024.09.0 --> 2024.11.0
  • numcodecs 0.13.1 --> 0.14.1

By the way satpy 0.52.1-2 was build successfully on 2024-11-04, so less than on month ago.

@djhoese
Copy link
Member

djhoese commented Dec 2, 2024

The AttributeErrors in Python 3.13 seem like something fsspec needs to fix (but CC @mraspaud and @pnuu just to be sure they see it). The value changes, I wonder if this is from the dtype work that @pnuu did. Would it be possible @avalentino to run the tests for satpy 0.52.1 with the current environment? Actually, regardless, I think the differences in the results are decently within 32-bit floating point precision, right?

@avalentino
Copy link
Contributor Author

satpy v0.52.1 currently only fails with Python 3.13:

FAILED satpy/tests/test_readers.py::TestFSFile::test_open_zip_fs_regular_filename
FAILED satpy/tests/test_readers.py::TestFSFile::test_sorting_fsfiles - Attrib...
FAILED satpy/tests/test_readers.py::TestFSFile::test_equality - AttributeErro...
FAILED satpy/tests/test_readers.py::TestFSFile::test_hash - AttributeError: '...
= 4 failed, 2455 passed, 2 skipped, 256 deselected, 6 xfailed, 580 warnings in 434.48s (0:07:14) =

@pnuu
Copy link
Member

pnuu commented Dec 4, 2024

For me the tests pass with up-to-date packages from conda-forge with Python 3.12 and Satpy 0.53.0:

$ conda create -y -n py312_test python=3.12 satpy=0.53.0 defusedxml pytest
$ pip install pytest_lazy_fixtures
$ pytest satpy/tests/test_modifiers.py::TestSunZenithCorrector
============================= test session starts ==============================
platform linux -- Python 3.12.7, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/lahtinep/Software/pytroll/pytroll_packages/satpy
configfile: pyproject.toml
plugins: lazy-fixtures-1.1.1
collected 13 items                                                             

satpy/tests/test_modifiers.py .............                              [100%]

============================== 13 passed in 0.54s ==============================

@avalentino
Copy link
Contributor Author

@pnuu please not that in Debian Sid we still have numpy v1.26 currently

@pnuu
Copy link
Member

pnuu commented Dec 4, 2024

Ooh, I see! Yeah, with numpy=1.26 I see the failures in the same two tests but with different values and number of elements:

======================================== FAILURES ========================================
__________ TestSunZenithCorrector.test_basic_default_provided[sunz_ds1-float32] __________

self = <satpy.tests.test_modifiers.TestSunZenithCorrector object at 0x774a9be83bc0>
data_arr = <xarray.DataArray 'ones_like-ce66f26c3a4b4f555c316e807cbe1781' (y: 2, x: 2)> Size: 32B
dask.array<ones_like, shape=(2,... test\nProjection ID: test\nProje...
    start_time:  2018-01-01 18:00:00
    modifiers:   ()
    name:        test_vis
sunz_sza = <xarray.DataArray 'rad2deg-fc1dde86ae8b0fac7523bd74851282df' (y: 2, x: 2)> Size: 32B
dask.array<rad2deg, shape=(2, 2),...x        (x) int64 16B 0 1
Attributes:
    area:     Area ID: test\nDescription: test\nProjection ID: test\nProjecti...
dtype = <class 'numpy.float32'>

    @pytest.mark.parametrize("dtype", [np.float32, np.float64])
    @pytest.mark.parametrize("data_arr", [lazy_fixture("sunz_ds1"), lazy_fixture("sunz_ds1_stacked")])
    def test_basic_default_provided(self, data_arr, sunz_sza, dtype):
        """Test default limits when SZA is provided."""
        from satpy.modifiers.geometry import SunZenithCorrector
        comp = SunZenithCorrector(name="sza_test", modifiers=tuple())
        res = comp((data_arr.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
        expected = np.array([[22.401667, 22.31777], [22.437503, 22.353533]], dtype=dtype)
        values = res.values
>       np.testing.assert_allclose(values, expected)

satpy/tests/test_modifiers.py:162: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<function assert_allclose.<locals>.compare at 0x774a9f1cbd80>, array([[22.401669, 22.317774],
       [22.437506, 22.353533]], dtype=float32), array([[22.401667, 22.31777 ],
       [22.437504, 22.353533]], dtype=float32))
kwds = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-07, atol=0', 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError: 
E           Not equal to tolerance rtol=1e-07, atol=0
E           
E           Mismatched elements: 1 / 4 (25%)
E           Max absolute difference: 3.8146973e-06
E           Max relative difference: 1.7092646e-07
E            x: array([[22.401669, 22.317774],
E                  [22.437506, 22.353533]], dtype=float32)
E            y: array([[22.401667, 22.31777 ],
E                  [22.437504, 22.353533]], dtype=float32)

../../../../mambaforge/envs/py312_test/lib/python3.12/contextlib.py:81: AssertionError
______ TestSunZenithCorrector.test_basic_default_provided[sunz_ds1_stacked-float32] ______

self = <satpy.tests.test_modifiers.TestSunZenithCorrector object at 0x774a9be83e30>
data_arr = <xarray.DataArray 'ones_like-ce66f26c3a4b4f555c316e807cbe1781' (y: 2, x: 2)> Size: 32B
dask.array<ones_like, shape=(2,...dAreaDefinition object at 0x774a9...
    start_time:  2018-01-01 18:00:00
    modifiers:   ()
    name:        test_vis
sunz_sza = <xarray.DataArray 'rad2deg-fc1dde86ae8b0fac7523bd74851282df' (y: 2, x: 2)> Size: 32B
dask.array<rad2deg, shape=(2, 2),...x        (x) int64 16B 0 1
Attributes:
    area:     Area ID: test\nDescription: test\nProjection ID: test\nProjecti...
dtype = <class 'numpy.float32'>

    @pytest.mark.parametrize("dtype", [np.float32, np.float64])
    @pytest.mark.parametrize("data_arr", [lazy_fixture("sunz_ds1"), lazy_fixture("sunz_ds1_stacked")])
    def test_basic_default_provided(self, data_arr, sunz_sza, dtype):
        """Test default limits when SZA is provided."""
        from satpy.modifiers.geometry import SunZenithCorrector
        comp = SunZenithCorrector(name="sza_test", modifiers=tuple())
        res = comp((data_arr.astype(dtype), sunz_sza.astype(dtype)), test_attr="test")
        expected = np.array([[22.401667, 22.31777], [22.437503, 22.353533]], dtype=dtype)
        values = res.values
>       np.testing.assert_allclose(values, expected)

satpy/tests/test_modifiers.py:162: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<function assert_allclose.<locals>.compare at 0x774a9f1cbd80>, array([[22.401669, 22.317774],
       [22.437506, 22.353533]], dtype=float32), array([[22.401667, 22.31777 ],
       [22.437504, 22.353533]], dtype=float32))
kwds = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-07, atol=0', 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError: 
E           Not equal to tolerance rtol=1e-07, atol=0
E           
E           Mismatched elements: 1 / 4 (25%)
E           Max absolute difference: 3.8146973e-06
E           Max relative difference: 1.7092646e-07
E            x: array([[22.401669, 22.317774],
E                  [22.437506, 22.353533]], dtype=float32)
E            y: array([[22.401667, 22.31777 ],
E                  [22.437504, 22.353533]], dtype=float32)

From my failures I'd set atol=1e-6 and rtol=1e-7, but those wouldn't be enough for the failures you get with Sid. Maybe 1e-5 and 1e-6 then?

I need to jump to task at work for the rest of the day. Unless someone has PR'd the tolerance changes I think I have time during the evening to do that.

@avalentino
Copy link
Contributor Author

TestFSFile: tests no longer fail with fsspec 2024.10.0.
Maybe you could update the dependency on fsspec as versioned dependency fsspec >= 2024.10.0.

@djhoese
Copy link
Member

djhoese commented Dec 9, 2024

I think @mraspaud (who is on vacation) wanted to add fsspec as a hard requirement for satpy. With that and this request for one of the most recent versions as a limit, that seems pretty strict/difficult for users I would think. Maybe that's just me though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants