Programmatic Access to Logs

[vlerenc]

What would you like to be added:
Please consider adding support to programmatically retrieve logs from VictoriaLogs (replacing Vali) without port-forwarding (not available to end users anyhow). That was always possible in the past, even though only unofficially, through the Grafana/Plutono data sources.

Security is of concern here, so I suggest to follow the way how we exposed Prometheus today to our end users (to not boil the ocean here as the whole VictoriaLogs roll-out is on a timer). With Prometheus, we have put everything behind basic auth and it's semi-officially exposed already to Gardener end users. All I am asking is to do the same for VictoriaLogs before Plutono (and its Vali data source) is gone. Of course, it would be nice to overhaul the way we have exposed Prometheus (and then VictoriaLogs), but there is no need to do it all in one go as Prometheus is already exposed this way for long. At a later point in time, both can be revamped - in the same way.

The important bit here is to not lose programmatic access to logs and preserve it in some shape of form, please.

Docs:

• Query API behind /select <- This is what we want
• Ingestion API behind /insert <- This is what we do not want

Our ingress gateway could expose the /select path behind basic auth, indirectly blocking every other path - just like with Prometheus.

Why is this needed:
Some operators prefer programmatic access, so that they can use CLI tools or script complex operations (I am one of them, but there are probably a few more).

Further use cases include but are not limited to automated (post-)processing of logs, downloading them for evidence/audits, cross-cluster log analysis, etc.

Yet another use case worth pointing out explicitly is the same as with Prometheus and why we exposed it directly: to federate logs (control plane and data plane), e.g. also possibly because of limited retention of logs (by time or data volume; a.k.a. flight recorder for later analysis), although this will be mitigated by observability 2.0.

Programatic (=API) access is something we should consider basic baseline functionality for any service (in this case retrieving logs from a shoot cluster control plane in Gardener).