Merge branch 'develop'

CS-SI · Mar 9, 2021 · 06d2306 · 06d2306
2 parents 0eb6153 + ee240fa
commit 06d2306
Show file tree

Hide file tree

Showing 82 changed files with 14,822 additions and 3,409 deletions.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -1,4 +1,4 @@
-image: python:3.5
+image: python:3.6
 
 stages:
   - build
@@ -23,16 +23,16 @@ build-doc:
     - if: $CI_COMMIT_BRANCH == 'master'   # Execute jobs when a new commit is pushed to master branch
     - if: $CI_MERGE_REQUEST_ID            # Execute jobs in merge request context
 
-py35:unit-tests:
+py36:unit-tests:
   stage: test
   script:
     - python -m pip install --upgrade pip
     - python -m pip install -r requirements-dev.txt
     - mkdir -p test-reports
     - |
       nosetests -v --cover-package=eodag\
-                    --with-xunit --xunit-file test-reports/eodag_test_report_py35.xml\
-                    --with-coverage --cover-xml --cover-xml-file test-reports/eodag_test_coverage_report_py35.xml\
+                    --with-xunit --xunit-file test-reports/eodag_test_report_py36.xml\
+                    --with-coverage --cover-xml --cover-xml-file test-reports/eodag_test_coverage_report_py36.xml\
                     --with-doctest\
                     --cover-erase\
                     --cover-inclusive\
@@ -41,8 +41,8 @@ py35:unit-tests:
   artifacts:
     when: always
     reports:
-      junit: test-reports/eodag_test_report_py35.xml
-      cobertura: test-reports/eodag_test_coverage_report_py35.xml
+      junit: test-reports/eodag_test_report_py36.xml
+      cobertura: test-reports/eodag_test_coverage_report_py36.xml
   rules:
     - if: $CI_COMMIT_BRANCH == 'master'   # Execute jobs when a new commit is pushed to master branch
     - if: $CI_MERGE_REQUEST_ID            # Execute jobs in merge request context

diff --git a/CHANGES.rst b/CHANGES.rst
@@ -1,6 +1,24 @@
 Release history
 ---------------
 
+2.1.0 (2021-03-09)
+++++++++++++++++++
+
+- `earth_search <https://www.element84.com/earth-search>`_ and
+  `usgs_satapi_aws <https://landsatlook.usgs.gov/sat-api>`_ as new providers
+- Updated ``HTTPDownload`` plugin, handling products with multiple assets
+- New plugin ``AwsAuth``, enables AWS authentication using no-sign-request, profile, ``~/.aws/*``
+- New search plugin ``StaticStacSearch`` and updated
+  `STAC client tutorial <https://eodag.readthedocs.io/en/latest/tutorials/tuto_stac_client.html>`_
+- New tutorial for `Copernicus DEM <https://eodag.readthedocs.io/en/latest/tutorials/tuto_cop_dem.html>`_
+- Remove ``unidecode`` dependency
+- Start/end dates passed to sobloo are now in UTC, and make it clear that search dates must be in UTC
+- Locations must now be passed to ``search()`` method as a dictionnary
+- Metadata mapping update and uniformization, fixes `GH-154 <https://github.com/CS-SI/eodag/issues/154>`_
+- Raise a ``ValueError`` when a location search doesn't match any record and add a new ``locations``
+  parameter to ``EODataAccessGateway.search``.
+- Drop support of Python 3.5
+
 2.0.1 (2021-02-05)
 ++++++++++++++++++
 
@@ -14,8 +32,9 @@ Release history
 - Allow to dynamically set download options, fixes `GH-145 <https://github.com/CS-SI/eodag/issues/145>`_ and
   `GH-112 <https://github.com/CS-SI/eodag/issues/112>`_
 - New tutorials for STAC and search by geometry, fixes `GH-139 <https://github.com/CS-SI/eodag/issues/139>`_
-- New crunches FilterDate, FilterProperty and updated FilterOverlap, fixes `GH-137 <https://github.com/CS-SI/eodag/issues/137>`_
-- Use jsonpath-ng instead of jsonpath-rw and pyjq, pyshp instead of fiona
+- New crunches ``FilterDate``, ``FilterProperty`` and updated ``FilterOverlap``, fixes
+  `GH-137 <https://github.com/CS-SI/eodag/issues/137>`_
+- Use ``jsonpath-ng`` instead of ``jsonpath-rw`` and ``pyjq``, ``pyshp`` instead of ``fiona``
 - Better wrong or missing credentials handling
 - Add warning for the total number of results returned by theia
 - Support regex query from locations configuration
@@ -26,10 +45,10 @@ Release history
 2.0b2 (2020-12-18)
 ++++++++++++++++++
 
-- New method deserialize_and_register, fixes `GH-140 <https://github.com/CS-SI/eodag/issues/140>`_
-- Load static stac catalogs as SearchResult
-- Search on unknown product types using GENERIC_PRODUCT_TYPE
-- get_data, drivers and rpc server moved to `eodag-cube <https://github.com/CS-SI/eodag-cube>`_
+- New method ``deserialize_and_register``, fixes `GH-140 <https://github.com/CS-SI/eodag/issues/140>`_
+- Load static stac catalogs as ``SearchResult``
+- Search on unknown product types using ``GENERIC_PRODUCT_TYPE``
+- ``get_data``, drivers and rpc server moved to `eodag-cube <https://github.com/CS-SI/eodag-cube>`_
 - Removed fixed dependencies, fixes `GH-82 <https://github.com/CS-SI/eodag/issues/82>`_
 - Use locations conf template by default
 

diff --git a/NOTICE b/NOTICE
@@ -78,16 +78,6 @@ the Django Project, licensed under the BSD-3-Clause Licence. Follow project link
 https://github.com/django/django/
 
 
-================================================================
-The GNU General Public License v2.0
-================================================================
-
-The following components are provided under the  BSD-3-Clause License  (https://opensource.org/licenses/GPL-2.0).
-See project link for details.
-
-https://github.com/avian2/unidecode
-
-
 ================================================================
 The Python-2.0 License
 ================================================================

diff --git a/README.rst b/README.rst
@@ -27,41 +27,140 @@ EODAG (Earth Observation Data Access Gateway) is a command line tool and a plugi
 aggregating results and downloading remote sensed images while offering a unified API for data access regardless of the
 data provider. The EODAG SDK is structured around three functions:
 
-    * List product types: list of supported products and their description
+* List product types: list of supported products and their description
 
-    * Search products (by product type or uid) : searches products according to the search criteria provided
+* Search products (by product type or uid) : searches products according to the search criteria provided
 
-    * Download products : download product “as is"
+* Download products : download product “as is"
 
 EODAG is developed in Python. It is structured according to a modular plugin architecture, easily extensible and able to
 integrate new data providers. Three types of plugins compose the tool:
 
-    * Catalog search plugins, responsible for searching data (OpenSearch, CSW, ...), building paths, retrieving quicklook,
-      combining results
+* Catalog search plugins, responsible for searching data (OpenSearch, CSW, ...), building paths, retrieving quicklook,
+  combining results
 
-    * Download plugins, allowing to download and retrieve data locally (via FTP, HTTP, ..), always with the same directory
-      organization
+* Download plugins, allowing to download and retrieve data locally (via FTP, HTTP, ..), always with the same directory
+  organization
 
-    * Authentication plugins, which are used to authenticate the user on the external services used (JSON Token, Basic Auth, OAUTH, ...).
+* Authentication plugins, which are used to authenticate the user on the external services used (JSON Token, Basic Auth, OAUTH, ...).
 
 Since v2.0 EODAG can be run as `STAC client or server <https://eodag.readthedocs.io/en/latest/intro.html#stac-client-and-server>`_.
 
 Read `the documentation <https://eodag.readthedocs.io/en/latest/>`_ for more insights.
 
-.. image:: docs/_static/eodag_stac_client.png
-   :alt: EODAG as STAC client
+.. image:: https://raw.githubusercontent.com/CS-SI/eodag/develop/docs/_static/eodag_overview.png
+   :alt: EODAG overview
    :class: no-scaled-link
 
 Installation
 ============
 
-EODAG is on `PyPI <https://pypi.org/project/eodag/>`_::
+EODAG is on `PyPI <https://pypi.org/project/eodag/>`_:
+
+.. code-block:: bash
 
     python -m pip install eodag
 
+
 Usage
 =====
 
+For downloading you will need to fill your credentials for the desired providers in your
+`eodag user configuration file <https://eodag.readthedocs.io/en/latest/intro.html#how-to-configure-authentication-for-available-providers>`_.
+The file will automatically be created with empty values on the first run.
+
+Python API
+----------
+
+Example usage for interacting with the api in your Python code:
+
+.. code-block:: python
+
+    from eodag import EODataAccessGateway
+
+    dag = EODataAccessGateway()
+    product_type = 'S2_MSI_L1C'
+    footprint = {'lonmin': 1, 'latmin': 43.5, 'lonmax': 2, 'latmax': 44}
+    start, end = '2021-01-01', '2021-01-15'
+    search_results, found_nb = dag.search(productType=product_type, geom=footprint, start=start, end=end)
+    product_paths = dag.download_all(search_results)
+    for path in product_paths:
+      print('Downloaded : {}'.format(path))
+
+
+STAC REST API
+-------------
+
+An eodag installation can be exposed through a STAC compliant REST api from the command line:
+
+.. code-block:: bash
+
+    $ eodag serve-rest --help
+    Usage: eodag serve-rest [OPTIONS]
+
+      Start eodag HTTP server
+
+    Options:
+      -f, --config PATH   File path to the user configuration file with its
+                          credentials
+      -d, --daemon TEXT   run in daemon mode
+      -w, --world         run flask using IPv4 0.0.0.0 (all network interfaces),
+                          otherwise bind to 127.0.0.1 (localhost). This maybe
+                          necessary in systems that only run Flask  [default:
+                          False]
+      -p, --port INTEGER  The port on which to listen  [default: 5000]
+      --debug             Run in debug mode (for development purpose)  [default:
+                          False]
+      --help              Show this message and exit.
+
+    # run server
+    $ eodag serve-rest
+
+    # list available product types for ``peps`` provider:
+    $ curl "http://127.0.0.1:5000/collections?provider=peps" | jq ".collections[].id"
+    "S1_SAR_GRD"
+    "S1_SAR_OCN"
+    "S1_SAR_SLC"
+    "S2_MSI_L1C"
+    "S2_MSI_L2A"
+    "S3_EFR"
+    "S3_ERR"
+    "S3_LAN"
+    "S3_OLCI_L2LFR"
+    "S3_OLCI_L2LRR"
+    "S3_SLSTR_L1RBT"
+    "S3_SLSTR_L2LST"
+
+    # search for items
+    $ curl "http://127.0.0.1:5000/search?collections=S2_MSI_L1C&bbox=0,43,1,44&datetime=2018-01-20/2018-01-25" \
+    | jq ".context.matched"
+    6
+
+    # browse for items
+    $ curl "http://127.0.0.1:5000/S2_MSI_L1C/country/FRA/year/2021/month/01/day/25/cloud_cover/10/items" \
+    | jq ".context.matched"
+    9
+
+    # get download link
+    $ curl "http://127.0.0.1:5000/S2_MSI_L1C/country/FRA/year/2021/month/01/day/25/cloud_cover/10/items" \
+    | jq ".features[0].assets.downloadLink.href"
+    "http://127.0.0.1:5000/S2_MSI_L1C/country/FRA/year/2021/month/01/day/25/cloud_cover/10/items/S2A_MSIL1C_20210125T105331_N0209_R051_T31UCR_20210125T130733/download"
+
+    # download
+    $ wget "http://127.0.0.1:5000/S2_MSI_L1C/country/FRA/year/2021/month/01/day/25/cloud_cover/10/items/S2A_MSIL1C_20210125T105331_N0209_R051_T31UCR_20210125T130733/download"
+
+
+You can also browse over your STAC API server using `STAC Browser <https://github.com/radiantearth/stac-browser>`_
+with ``CATALOG_URL=http://127.0.0.1:5000`` :
+
+.. image:: https://raw.githubusercontent.com/CS-SI/eodag/develop/docs/_static/stac_browser_example_600.png
+   :target: https://raw.githubusercontent.com/CS-SI/eodag/develop/docs/_static/stac_browser_example.png
+   :alt: STAC browser example
+   :width: 600px
+
+
+For more information, see `STAC REST interface usage <https://eodag.readthedocs.io/en/latest/use.html#stac-rest-interface>`_.
+
 Command line interface
 ----------------------
 
@@ -117,49 +216,6 @@ of this bbox"
   verbose the tool is. For a full verbose output, do for example: ``eodag -vvv list``
 
 
-STAC REST API
--------------
-
-An eodag installation can be exposed through a STAC compliant REST api from the command line::
-
-    # eodag serve-rest --help
-    Usage: eodag serve-rest [OPTIONS]
-
-      Start eodag HTTP server
-
-    Options:
-      -f, --config PATH   File path to the user configuration file with its
-                          credentials
-      -d, --daemon TEXT   run in daemon mode
-      -w, --world         run flask using IPv4 0.0.0.0 (all network interfaces),
-                          otherwise bind to 127.0.0.1 (localhost). This maybe
-                          necessary in systems that only run Flask  [default:
-                          False]
-      -p, --port INTEGER  The port on which to listen  [default: 5000]
-      --debug             Run in debug mode (for development purpose)  [default:
-                          False]
-      --help              Show this message and exit.
-
-
-Python API
-----------
-
-Example usage for interacting with the api in your Python code:
-
-.. code-block:: python
-
-    from eodag import EODataAccessGateway
-
-    dag = EODataAccessGateway()
-    product_type = 'S2_MSI_L1C'
-    footprint = {'lonmin': 1, 'latmin': 43.5, 'lonmax': 2, 'latmax': 44}
-    start, end = '2021-01-01', '2021-01-15'
-    search_results = dag.search(productType=product_type, geom=footprint, start=start, end=end)
-    product_paths = dag.download_all(search_results)
-    for path in product_paths:
-        print('Downloaded : {}'.format(path))
-
-
 Contribute
 ==========
 
@@ -194,21 +250,21 @@ To run the entire tests (units, integration and end-to-end)::
     * When contributing to tutorials, you will need to keep notebook outputs
       and save widget state. Otherwise outputs will not be visible in documentation.
 
-    * eodag is tested against python versions 3.5, 3.6, 3.7, 3.8 and 3.9. Ensure you have
+    * eodag is tested against python versions 3.6, 3.7, 3.8 and 3.9. Ensure you have
       these versions installed before you run tox. You can use
       `pyenv <https://github.com/pyenv/pyenv>`_ to manage many different versions
       of python
 
 Releases are made by tagging a commit on the master branch. To make a new release,
 
-    * Ensure you correctly updated `README.rst` and `CHANGES.rst` (and occasionally,
-      also `NOTICE` - in case a new dependency is added).
-    * Check that the version string in `eodag/__meta__.py` (the variable `__version__`)
-      is correctly updated
-    * Push your local master branch to remote.
-    * Tag the commit that represents the state of the release with a message. For example,
-      for version 1.0, do this: `git tag -a v1.0 -m 'version 1.0'`
-    * Push the tags to github: `git push --tags`.
+* Ensure you correctly updated `README.rst` and `CHANGES.rst` (and occasionally,
+  also `NOTICE` - in case a new dependency is added).
+* Check that the version string in `eodag/__meta__.py` (the variable `__version__`)
+  is correctly updated
+* Push your local master branch to remote.
+* Tag the commit that represents the state of the release with a message. For example,
+  for version 1.0, do this: `git tag -a v1.0 -m 'version 1.0'`
+* Push the tags to github: `git push --tags`.
 
 The documentation is managed by a webhook, and the latest documentation on readthedocs follows
 the documentation present in `master`. Therefore, there is nothing to do apart from updating

diff --git a/bitbucket-pipelines.yml b/bitbucket-pipelines.yml
diff --git a/docs/_static/eodag_overview.png b/docs/_static/eodag_overview.png