Linkerd Metrics - add configuration for authority label

[marcelovcpereira]

Hello, we had some discussions (here and here) in linkerd slack about the impact that the label "authority" can cause in prometheus in some situations.

Because of that we are thinking about contributing to linkerd2-proxy.
Do you think it would be valuable to have a configuration to define the granularity of the metric label authority ?
We could have an option to have there the full qualified domain name (default) or only the top level domain.
The top level domain option would be useful to reduce cardinality of the timeseries in prometheus in case we have too many subdomains (as in our use case - per customer, which can reach multiple thousands).

[olix0r]

@marcelovcpereira Is it possible to manage this cardinality as part of the prometheus scrape config?

[marcelovcpereira]

@olix0r Unfortunately not. I mean, not at least if we want to keep data consistent.

Prometheus has a feature that could be used to decrease the label cardinality (and we have tried it): label drop.
It actually worked (regarding reducing memory pressure on Prometheus), but the problem is that the end result should not be duplicated timeseries. Here is the official statement:

[...] Care must be taken with labeldrop and labelkeep to ensure that metrics are still uniquely labeled once the labels are removed

So basically prometheus can "forget" the authority label, but it will result in multiple timeseries having the same signature, which would make only one of them persisted. We noticed a drastic drop in request rate via linkerd-grafana as well other metrics which use this label as dimension.