Fix allowing customisation of CREODIAS S3 eodata service details and …

…credentials

deploy/eoepca/data-access.sh

@@ -47,7 +47,7 @@ deployService() {
     serviceValues | helm ${ACTION_HELM} data-access data-access -f - \
       --repo https://eoepca.github.io/helm-charts \
       --namespace ${NAMESPACE} --create-namespace \
-      --version 1.4.0
+      --version 1.4.1
   fi
 }
 
@@ -61,6 +61,10 @@ global:
     AWS_HTTPS: "FALSE"
     startup_scripts:
       - /registrar_pycsw/registrar_pycsw/initialize-collections.sh
+    CREODIAS_EODATA_S3_ENDPOINT: "${CREODIAS_EODATA_S3_ENDPOINT}"
+    CREODIAS_EODATA_S3_ACCESS_KEY: "${CREODIAS_EODATA_S3_ACCESS_KEY}"
+    CREODIAS_EODATA_S3_ACCESS_SECRET: "${CREODIAS_EODATA_S3_ACCESS_SECRET}"
+    CREODIAS_EODATA_S3_REGION: "${CREODIAS_EODATA_S3_REGION}"
 
   # The data-access relies on the value 'ingress.tls.hosts[0]' to specify the service
   # hostname. So this must be supplied even if the ingress is disabled.
@@ -83,9 +87,9 @@ global:
     data:
       data:
         type: S3
-        endpoint_url: http://data.cloudferro.com
-        access_key_id: access
-        secret_access_key: access
+        endpoint_url: "${CREODIAS_EODATA_S3_ENDPOINT}"
+        access_key_id: "${CREODIAS_EODATA_S3_ACCESS_KEY}"
+        secret_access_key: "${CREODIAS_EODATA_S3_ACCESS_SECRET}"
         region_name: RegionOne
         validate_bucket_name: false
     cache:

deploy/eoepca/eoepca-options

@@ -85,6 +85,10 @@ if [ -z "${_EOEPCA_OPTIONS_}" ]; then
 
   # Data
   export CREODIAS_DATA_SPECIFICATION="${CREODIAS_DATA_SPECIFICATION:-false}"
+  export CREODIAS_EODATA_S3_ENDPOINT="${CREODIAS_EODATA_S3_ENDPOINT:-http://data.cloudferro.com}"
+  export CREODIAS_EODATA_S3_ACCESS_KEY="${CREODIAS_EODATA_S3_ACCESS_KEY:-access}"
+  export CREODIAS_EODATA_S3_ACCESS_SECRET="${CREODIAS_EODATA_S3_ACCESS_SECRET:-access}"
+  export CREODIAS_EODATA_S3_REGION="${CREODIAS_EODATA_S3_REGION:-RegionOne}"
 
   # Identity Service
   export TEMP_FORWARDING_PORT="${TEMP_FORWARDING_PORT:-9876}"

docs/index.md

@@ -6,6 +6,7 @@
 
     The following provides a summary of changes since the last release (v1.4)...
 
+    * **03/04/2024** - Update Data Access to chart `1.4.1` to introduce variables to remedy hard-coded harvester values for access to Creodias eodata. Ref. - `CREODIAS_EODATA_S3_ENDPOINT`, `CREODIAS_EODATA_S3_ACCESS_KEY`, `CREODIAS_EODATA_S3_ACCESS_SECRET` and `CREODIAS_EODATA_S3_REGION`.
     * **20/03/2024** - Correction to chart path for helm deployment of `eoepca-portal`
     * **20/03/2024** - Correct hardcoded OAuth client secret for Application Hub
     * **20/03/2024** - Clarify Gatekeeper encryption key must be 16 or 32 characters long

docs/quickstart/creodias-deployment.md

@@ -15,6 +15,10 @@ The CREODIAS deployment applies the following configuration:
 
 With reference to the file `creodias-options`, particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment...
 
+* Values for access to CREODIAS eodata...
+    * CREODIAS_EODATA_S3_ENDPOINT - if different from the default `http://data.cloudferro.com`
+    * Credentials required for the new clouds, including `WAW3-2` and `FRA1-2`<br>
+      Credentials must be created at - [https://eodata-keymanager.creodias.eu/panel/s3-credentials](https://eodata-keymanager.creodias.eu/panel/s3-credentials) - and set into the variables `CREODIAS_EODATA_S3_ACCESS_KEY` and `CREODIAS_EODATA_S3_ACCESS_SECRET`
 * Passwords: `MINIO_ROOT_PASSWORD`, `HARBOR_ADMIN_PASSWORD`
 * Identity Service credentials - e.g. `IDENTITY_SERVICE_DEFAULT_SECRET`, `IDENTITY_SERVICE_ADMIN_PASSWORD`, etc.
 * OpenStack details: see section [Openstack Configuration](scripted-deployment.md#openstack-configuration)
@@ -37,9 +41,11 @@ kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harve
 
 See the [Harvester](#harvester) section below for an explanation of this harvester configuration.
 
+See [EOData Catalogue API Manual on CREODIAS](https://creodias.docs.cloudferro.com/en/latest/eodata/EOData-Catalogue-API-Manual-on-Creodias.html) for details regarding access to the CREODIAS data offering.
+
 ## Data Specification Walkthrough
 
-The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option `CREODIAS_DATA_SPECIFICATION=true` - see [Environment Variables](scripted-deployment.md#environment-variables).
+The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option `CREODIAS_DATA_SPECIFICATION=true` - see [Environment Variables](scripted-deployment.md#environment-variables). In addition, it may be necessary to set the variable `CREODIAS_EODATA_S3_ENDPOINT` if different from the default - for example the value `http://eodata.cloudferro.com` for the `WAW3-2` Cloudferro cloud.
 
 This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example.
 

docs/quickstart/data-access-deployment.md

@@ -14,7 +14,8 @@ The data-access deployment applies the following configuration:
 * Services deployed:
     * Resource Catalogue for data discovery
     * Data Access for data visualisation and download
-* Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the `eodata` network - [see description of variable `CREODIAS_DATA_SPECIFICATION`](scripted-deployment.md#environment-variables)
+* Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the `eodata` network - [see description of variable `CREODIAS_DATA_SPECIFICATION`](scripted-deployment.md#environment-variables)<br>
+  _Note that it may be necessary to set the variable `CREODIAS_EODATA_S3_ENDPOINT` if different from the default - for example the value `http://eodata.cloudferro.com` for the `WAW3-2` Cloudferro cloud._
 * Open ingress are enabled for unauthenticated access to resource-catalogue and data-access services
 * Other eoepca services not deployed
 

docs/quickstart/exploitation-deployment.md

@@ -17,7 +17,8 @@ The exploitation deployment applies the following configuration:
     * Data Access for data visualisation and download
     * Minio for S3 object storage
 * ADES stage-out to Minio
-* Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the `eodata` network - [see description of variable `CREODIAS_DATA_SPECIFICATION`](scripted-deployment.md#environment-variables)
+* Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the `eodata` network - [see description of variable `CREODIAS_DATA_SPECIFICATION`](scripted-deployment.md#environment-variables)<br>
+  _Note that it may be necessary to set the variable `CREODIAS_EODATA_S3_ENDPOINT` if different from the default - for example the value `http://eodata.cloudferro.com` for the `WAW3-2` Cloudferro cloud._
 * Open ingress are enabled for unauthenticated access to ADES, resource-catalogue and data-access services
 * Other eoepca services not deployed
 

docs/quickstart/scripted-deployment.md

@@ -68,6 +68,10 @@ The script [`deploy/eoepca/eoepca.sh`](https://github.com/EOEPCA/deployment-guid
     **STAGEOUT_TARGET** | Configures the ADES with the destination to which it should push processing results:<br>`workspace` - via the Workspace API<br>`minio` - to minio S3 object storage | `workspace`
     **INSTALL_FLUX** | The Workspace API relies upon [Flux CI/CD](https://fluxcd.io/), and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value `false` to suppress the Workspace API flux install | `true`
     **CREODIAS_DATA_SPECIFICATION** | Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services.<br>_Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the `eodata` network._ | `false`
+    **CREODIAS_EODATA_S3_ENDPOINT** | URL for the S3 endpoint in CREODIAS | `http://data.cloudferro.com`
+    **CREODIAS_EODATA_S3_ACCESS_KEY** | Access key for CREODIAS S3 endpoint | `access`
+    **CREODIAS_EODATA_S3_ACCESS_SECRET** | Access secret for CREODIAS S3 endpoint | `access`
+    **CREODIAS_EODATA_S3_REGION** | Region for the S3 endpoint in CREODIAS | `RegionOne`
     **TEMP_FORWARDING_PORT** | Local port used during the scripted deployment for `kubectl port-forward` operations | `9876`
 
 ### Command-line Arguments