Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CLOUD_COVERAGE not found in metadata #156

Closed
jteulade opened this issue Jun 10, 2024 · 4 comments · Fixed by #157
Closed

CLOUD_COVERAGE not found in metadata #156

jteulade opened this issue Jun 10, 2024 · 4 comments · Fixed by #157
Labels
bug Something isn't working

Comments

@jteulade
Copy link
Contributor

For a lot of products whose : s3://sertit-projects-archives/projects/2017/AEAG/mathias_AEAG/AEAG/FTP2/FC_41426_PRIO_SO13005352-3-06_DS_PHR1B_201307221050531_FR1_PX_E000N43_1103_01712.zip

I get this error:

Traceback (most recent call last):
  File "E:\JTeulade\repositories\stac\scripts\add_items.py", line 35, in <module>
    main()
  File "E:\JTeulade\repositories\stac\libs\sertit-utils\sertit\s3.py", line 99, in s3_env_wrapper
    return function(*_args, **_kwargs)
  File "E:\JTeulade\repositories\stac\scripts\add_items.py", line 22, in main
    add_items(valid_path_products, S3_TARGET_PATH, utils.get_ids())
  File "E:\JTeulade\repositories\stac\stac\stac.py", line 294, in add_items
    path_products = create_items(valid_path_products, catalog, s3_target_path)
  File "E:\JTeulade\repositories\stac\stac\stac.py", line 205, in create_items
    create_item(
  File "E:\JTeulade\repositories\stac\stac\stac.py", line 132, in create_item
    prod.stac.create_item()
  File "e:\jteulade\repositories\eoreader\eoreader\products\product.py", line 1424, in stac
    self._stac = StacItem(self)
  File "e:\jteulade\repositories\eoreader\eoreader\stac\stac_item.py", line 65, in __init__
    self.eo = EoExt(prod, **kwargs)
  File "e:\jteulade\repositories\eoreader\eoreader\stac\stac_extensions.py", line 79, in __init__
    self.cloud_cover = prod.get_cloud_cover()
  File "C:\Users\jteulade\AppData\Local\anaconda3\envs\arcgispro-eo\lib\site-packages\methodtools.py", line 72, in __call__
    return self.__call__(*args, **kwargs)
  File "e:\jteulade\repositories\eoreader\eoreader\__init__.py", line 34, in wrapper
    return func(*args, **kwargs)
  File "e:\jteulade\repositories\eoreader\eoreader\products\optical\dimap_v2_product.py", line 1229, in get_cloud_cover
    raise InvalidProductError("CLOUD_COVERAGE not found in metadata!")
eoreader.exceptions.InvalidProductError: CLOUD_COVERAGE not found in metadata!

We can't create a STAC item.
A solution can be to catch the InvalidProductError when we try to get the cloud cover here
@jteulade jteulade added the bug Something isn't working label Jun 10, 2024
jteulade added a commit that referenced this issue Jun 10, 2024
@jteulade
#156: catch InvalidProductError when we try to get cloud_coverage pro…
1106de5 
…perty
@jteulade jteulade linked a pull request Jun 10, 2024 that will close this issue
#156: catch InvalidProductError when we try to get cloud_coverage pro… #157
Merged
2 tasks
@remi-braun
Copy link
Member

The better solution is to find the CLOUD COVER key in these product metadata and update the function accordingly 🙏

@jteulade
Copy link
Contributor Author

I don't find CLOUD COVER neither..
I have these properties:

<Imaging_Quality_Measurement>
      <QUALITY_TABLES>PHR</QUALITY_TABLES>
      <MEASURE_NAME>Cloud_Cotation (CLD)</MEASURE_NAME>
      <MEASURE_TYPE>AUTOMATIC</MEASURE_TYPE>
      <Quality_Mask>
        <Component>
          <COMPONENT_TITLE>Cloud_Cotation Mask</COMPONENT_TITLE>
          <COMPONENT_CONTENT>GML MASK</COMPONENT_CONTENT>
          <COMPONENT_TYPE>ENCAPSULATED</COMPONENT_TYPE>
          <COMPONENT_PATH href="MASKS/CLD_PHR1B_P_201307221050531_SEN_693384101-001_MSK.GML"/>
        </Component>
      </Quality_Mask>
    </Imaging_Quality_Measurement>

@remi-braun
Copy link
Member

Ok then in this case set 0.0 as per the default function :)

@jteulade
Copy link
Contributor Author

Same problem for

<class 'eoreader.exceptions.InvalidProductError'>: s3://sertit-rms2/archives/rms/2016/EMSN-028-Loiret/EMSN028-AOI-01-FRANCE/02IMAGES/POST/RAW/20160609_de2/DE2_PSH_L1B_000000_20160609T103920_20160609T103923_DE2_10692_0F40.zip -> SPACEMETRIC:SENSOR_AZIMUTH, INCIDENCE_ANGLE or VIEWING_ANGLE not found in metadata!

Traceback (most recent call last):
  File "E:\JTeulade\repositories\stac\scripts\add_items.py", line 35, in <module>
    main()
  File "E:\JTeulade\repositories\stac\libs\sertit-utils\sertit\s3.py", line 99, in s3_env_wrapper
    return function(*_args, **_kwargs)
  File "E:\JTeulade\repositories\stac\scripts\add_items.py", line 22, in main
    add_items(valid_path_products, S3_TARGET_PATH, utils.get_ids())
  File "E:\JTeulade\repositories\stac\stac\stac.py", line 325, in add_items
    path_products = create_items(valid_path_products, catalog, s3_target_path)
  File "E:\JTeulade\repositories\stac\stac\stac.py", line 230, in create_items
    create_item(
  File "E:\JTeulade\repositories\stac\stac\stac.py", line 135, in create_item
    prod.stac.create_item()
  File "e:\jteulade\repositories\eoreader\eoreader\products\product.py", line 1424, in stac
    self._stac = StacItem(self)
  File "e:\jteulade\repositories\eoreader\eoreader\stac\stac_item.py", line 73, in __init__
    self.view = ViewExt(prod, **kwargs)
  File "e:\jteulade\repositories\eoreader\eoreader\stac\stac_extensions.py", line 345, in __init__
    view_az, off_nadir, incidence_angle = prod.get_mean_viewing_angles()
  File "C:\Users\jteulade\AppData\Local\anaconda3\envs\arcgispro-eo\lib\site-packages\methodtools.py", line 72, in __call__
    return self.__call__(*args, **kwargs)
  File "e:\jteulade\repositories\eoreader\eoreader\__init__.py", line 34, in wrapper
    return func(*args, **kwargs)
  File "e:\jteulade\repositories\eoreader\eoreader\products\optical\dimap_v1_product.py", line 310, in get_mean_viewing_angles
    raise InvalidProductError(
eoreader.exceptions.InvalidProductError: SPACEMETRIC:SENSOR_AZIMUTH, INCIDENCE_ANGLE or VIEWING_ANGLE not found in metadata!

jteulade added a commit that referenced this issue Jun 13, 2024
@jteulade
#156: catch all InvalidProductError due to properties not found in me…
cf2e8f3 
…tadata
jteulade added a commit that referenced this issue Jun 17, 2024
@jteulade
#156: catch InvalidProductError when we try to get cloud_coverage pro…
902d8d5 
…perty
jteulade added a commit that referenced this issue Jun 17, 2024
@jteulade
#156: catch all InvalidProductError due to properties not found in me…
f4ead6f 
…tadata
@jteulade jteulade mentioned this issue Jun 17, 2024
Closed
pixalytics added a commit to pixalytics-ltd/eoreader that referenced this issue Aug 12, 2024
@pixalytics @jteulade
@pre-commit-ci @remi-braun
upstream updates (#1)
f63d6d6 
* Fix sertit#140: replace QB satellite ID with QB02 satellite ID

* Fix sertit#142: catch StopIteration and return InvalidProductError

* Fix sertit#142: catch StopIteration in several products

* Fix sertit#142: raise InvalidProductError instead of log

* Fix sertit#145: remove nested=-1 for SPOT constellations, use the default nested=0 instead

* Fix sertit#145: update CHANGES.md

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix sertit#142: catch StopIteration and return InvalidProductError

* Avoid KeyError if a product metadata doesn't have a namespace

* sertit#142: catch StopIteration and return InvalidProductError

* sertit#156: catch InvalidProductError when we try to get cloud_coverage property

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* sertit#156: catch all InvalidProductError due to properties not found in metadata

* sertit#161: return an error if SPECTRAL_PROCESSING not found in metadata

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/PyCQA/flake8: 7.0.0 → 7.1.0](PyCQA/flake8@7.0.0...7.1.0)

* FIX: Handle RCM and RS2 products that doesn't bundle their extent in a KML file sertit#155

* OPTIM: Search correctly nested metadata in the Reader (without accidentally using a recursive glob)

* sertit#152: add SGF in RS2 product type

* FIX: Fix error in looking for bands in Sentinel-2 L1C archived products sertit#168

* FIX: Handle wrongly recognized Planet products because of the recursive nested mtd in the Reader sertit#169

* FIX: Force the loading of `DimapV1` bands in `float32`

* FIX: Handle the case where fiona isn't installed anymore (with `geopandas 1.0`)

* FIX: Don't make `pystac` a mandatory requirement

* Use sertit.types.is_iterable

* Use sertit.types.make_iterable

* Correct typo

* FIX: Fix an unknown Planet bug that just appeared (`'...Path' has no len()`)

* CI: Remove for now end-to-end tests with Python 3.11 and 3.10.

* v0.21.2

* INSTALL: Remove `pystac[validation]` (as it is an optional dependency) from setup.py, and create a `stac` extra feature.

* Updates in CHANGES.md

* v0.21.2

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1)
- [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0)

* FIX: Get better window name (if available) when writing bands on disk (in tmp folder) + update sertit's version to benefit to the vector's name

---------

Co-authored-by: jteulade <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Rémi Braun <[email protected]>
Co-authored-by: BRAUN REMI <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#156: catch InvalidProductError when we try to get cloud_coverage pro… sertit/eoreader
2 participants
@jteulade @remi-braun