How Linkerd Viz uses Prometheus

[channyein87]

I'm setting up Linkerd Viz for production and would like to understand how it uses Prometheus. Does Viz really call Prometheus to get metrics to show the RPS and success rates?

[wmorgan]

Yes

[channyein87]

Thanks got it. So web calls metric api and metric api uses prometheus. 👍🏻
https://github.com/linkerd/linkerd2/blob/main/web/main.go#L31
https://github.com/linkerd/linkerd2/blob/main/viz/metrics-api/cmd/main.go#L27

[MA3CIN]

@wmorgan @channyein87 Any chance you guys have the full queries that linkerd uses to get that data from Prometheus?
I can't figure out how to get the success rate and the p99 for latency

[wmorgan]

@MA3CIN The queries are generated programmatically through a whole bunch of code. One quick option would be to look in the Prometheus query logs and see what was executed. Another option would be to add some printfs to metrics-api/prometheus.go (or a similar function) to see which queries were sent over.

[kflynn]

@MA3CIN You can also check out https://buoyant.io/service-mesh-academy/metrics-dashboards-and-charts-oh-my, where we talk a little bit about this (in particular, we talk about the success rate metric). And you can look at what the published Grafana dashboard uses for P99.