Provide more metrics

[j08lue]

The Data Access BB can provide additional metrics apart from service health / liveliness. These could be performance metrics like response time and usage metrics like collections and items (STAC API) or tiles requested (TiTiler).

The approach we take to expose these metrics should be standardized among building blocks. In a discussion at progress review today, it sounded like there are already ideas / opinions about this, like a standard /metrics endpoint per building block or so.

Acceptance criteria

• Define which additional metrics are most valuable to service operators
• Implement metrics using OpenTelemetry in EOEPCA runtimes
• Surface the traces in the eoAPI Grafana dashboard
• Define how external services can collect these metrics, e.g. for central platform operations - align with other building blocks.
• Document the approach in eoapi-k8s

[j08lue]

We should probably also review, which metrics can rather be gathered by an API gateway.

For example

1. Number of requests per endpoint and method (GET, POST, etc.)
2. Service response times

Something better/only collected by the service is perhaps

1. Volume of data read (a proxy won't know)
2. Number of requests by STAC collection - probably would need to omit user-private collections, though, so maybe more a metric to collect in some STAC auth proxy we have recently been envisioning

[j08lue]

Inspiration re relevant metrics about an EO data hub (public and for operations):

• CDSE Dashboard: https://dashboard.dataspace.copernicus.eu/#/service-health
• Terrascope Dashboard: https://dashboards.terrascope.be/status/d/KlitTRUWz/terrascope-overview

(The eoAPI Grafana currently lives here: https://eoapisupport.develop.eoepca.org/, but is not available publicly yet #112)