PoC of Observability Platform API #3820
Assignees
Labels
team/atlas
Team Atlas
Comments
|
After testing various iterations of the ingresses, I was able to make the following setup work (without authentication for now) :
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-giantswarm
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
labels:
name: observability-platform-endpoint
namespace: loki
spec:
ingressClassName: nginx
rules:
- host: observability.golem.gaws.gigantic.io
http:
paths:
- backend:
service:
name: loki-read
port:
number: 3100
path: /logs(/|$)(.*)
pathType: ImplementationSpecific
tls:
- hosts:
- observability.golem.gaws.gigantic.io
secretName: observability-platform-endpoint-ingress-cert
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-giantswarm
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/use-regex: "true"
labels:
name: observability-platform-endpoint
namespace: mimir
spec:
ingressClassName: nginx
rules:
- host: observability.golem.gaws.gigantic.io
http:
paths:
- backend:
service:
name: mimir-query-frontend
port:
number: 8080
- backend:
service:
name: mimir-ruler
port:
number: 8080
path: /prometheus/api/v1/rules
pathType: ImplementationSpecific
tls:
- hosts:
- observability.golem.gaws.gigantic.io
secretName: observability-platform-endpoint-ingress-cert
Concerning Loki, eveything is going smoothly and I can query MC logs from my grafana deployed on a WC so only authentication is missing there. As for Mimir, I'm facing an issue that prevents the datasource to work as intended. I had to put an additional backend to allow access to the ruler as I noticed by looking at the I looked at the requests going through nginx after executing some queries from the MC's grafana, and they look quite different from the WC's ones :
|
|
Update concerning the current setup : now the ingresses are defined as such : apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-giantswarm
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/auth-url: https://oidc.golem.gaws.gigantic.io/oauth2/dex/auth
nginx.ingress.kubernetes.io/auth-signin: https://oidc.golem.gaws.gigantic.io/oauth2/dex/start?rd=$escaped_request_uri
labels:
name: observability-platform-endpoint
namespace: loki
spec:
ingressClassName: nginx
rules:
- host: observability.golem.gaws.gigantic.io
http:
paths:
- backend:
service:
name: loki-gateway
port:
number: 80
path: /loki/api/v1/query
pathType: ImplementationSpecific
- backend:
service:
name: loki-gateway
port:
number: 80
path: /loki/api/v1/labels
pathType: ImplementationSpecific
- backend:
service:
name: loki-gateway
port:
number: 80
path: /loki/api/v1/label
pathType: ImplementationSpecific
- backend:
service:
name: loki-gateway
port:
number: 80
path: /loki/api/v1/query_range
pathType: ImplementationSpecific
- backend:
service:
name: loki-gateway
port:
number: 80
path: /loki/api/v1/index
pathType: ImplementationSpecific
- backend:
service:
name: loki-gateway
port:
number: 80
path: /loki/api/v1/series
pathType: ImplementationSpecific
tls:
- hosts:
- observability.golem.gaws.gigantic.io
secretName: observability-platform-endpoint-ingress-cert
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-giantswarm
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/auth-url: https://oidc.golem.gaws.gigantic.io/oauth2/dex/auth
nginx.ingress.kubernetes.io/auth-signin: https://oidc.golem.gaws.gigantic.io/oauth2/dex/start?rd=$escaped_request_uri
labels:
name: observability-platform-endpoint
namespace: mimir
spec:
ingressClassName: nginx
rules:
- host: observability.golem.gaws.gigantic.io
http:
paths:
- backend:
service:
name: mimir-gateway
port:
number: 80
path: /prometheus/api/v1/labels
pathType: ImplementationSpecific
- backend:
service:
name: mimir-gateway
port:
number: 80
path: /prometheus/api/v1/label
pathType: ImplementationSpecific
- backend:
service:
name: mimir-gateway
port:
number: 80
path: /prometheus/api/v1/rules
pathType: ImplementationSpecific
- backend:
service:
name: mimir-gateway
port:
number: 80
path: /prometheus/api/v1/query
pathType: ImplementationSpecific
- backend:
service:
name: mimir-gateway
port:
number: 80
path: /prometheus/api/v1/query_exemplars
pathType: ImplementationSpecific
- backend:
service:
name: mimir-gateway
port:
number: 80
path: /prometheus/api/v1/status
pathType: ImplementationSpecific
- backend:
service:
name: mimir-gateway
port:
number: 80
path: /prometheus/api/v1/metadata
pathType: ImplementationSpecific
tls:
- hosts:
- observability.golem.gaws.gigantic.io
secretName: observability-platform-endpoint-ingress-certSo each path is hard-coded as discussed in the investigation issue. With this setup, I'm still having exactly the same issues described in my previous comment though. |
|
So after pairing with @QuentinBisson we discovered that the reason the metrics requests were failing was because of the Moreover, we finally managed to make the POC work with authentication using dex on
In the end, everything is working fine : one can log into the WC grafana and query observability data (whether logs or metrics) from the MC |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Motivation
with #3783 we found a concept for the Observability API - but now we also have to test that this concept is feasible. For this we can setup the API in one cluster and use it through a Grafana living in another cluster, while using our dex as sso. This will mirror the customer setup and we'll learn how to work with the API.
Todo
Keep in mind that this is a proof-of-concept - we are not yet interested in the perfect solution, but interested in learning, so fake it until we make it and cut corners where possible. We will have future implementation stories coming up once we proven that the concept works.
Outcome
The text was updated successfully, but these errors were encountered: