Promethus Exemplar Compatibility

[Aneurysm9]

Prometheus has decided to support exemplars on all metric types. This is not in compliance with the OpenMetrics spec and presents compatibility concerns given that our compatibility requirements align with OpenMetrics and Collector compatibility testing breaks with the latest Prometheus RC. Additionally, the SummaryDataPoint type in OTLP does not include support for exemplars, which will prevent successfully converting metrics from Prometheus to OTLP and back.

[fstab]

Hi Anthony, Fabian here, I'm the person who proposed supporting Exemplars on more time series at the Prometheus dev summit. Sorry that this is causing issues, I wasn’t expecting this to break anything, as it’s a backwards-compatible change.

The Prometheus team has been trying to open up a few things recently beyond what’s originally defined in OpenMetrics, like implementing native histograms, or exploring support for dots in label names. These ideas are being discussed and aligned with the OpenTelemetry community, and I feel these changes bring Prometheus and OpenTelemetry closer together.

I was hoping that allowing Exemplars on all time series goes in a similar direction. OpenTelemetry supports Exemplars on all time series (except for the deprecated Summaries). Most importantly, it supports Exemplars on Gauges. Until now these Exemplars got dropped when OpenTelemetry Gauges were stored in Prometheus. With the change, these Exemplars are supported by Prometheus as well, which is a good thing, as it brings OpenTelemetry metrics and Prometheus metrics closer together.

Regarding the OpenMetrics spec: I think you are right that we could have been more careful. Looking at the details of the spec it’s not clear to me whether the change is compliant with OpenMetrics or not (the ABNF allows Exemplars on all time series, and the data model says the ABNF must take precedence). However, this should at least have required some more discussion.

What do you think of having a call with me and @RichiH (editor of the OpenMetrics spec) to find a way forward that works for both the Prometheus and the OTel community?

[jsuereth]

@fstab Please include me in future communications. I think having a call between Prometheus / OpenMetrics / OTEL would be good. I don't see any reason why we can't add exemplars to Summary metrics for prometheus compatibility.

Also, I'm glad to see support expanded for other metric types!

[codeboten]

Has there been any progress on this issue? Seeing as it's been a few months since it was raised

[jsuereth]

Revisiting this issue.

• It appears like the only "breaking" change was a compliance test were we were attempting to enforce that sending exemplars fails on prometheus (i.e. a negative test). AFAICT, this never would have broken users of OpenTelemetry sending exemplars to prometheus.
• Regarding Exemplars themselves, we can just update the Prometheus<->OpenTelemetry specification here to resolve the issue.

I do not think we want to expand Summary to support exemplars, but if we see demand from the Prometheus community we can entertain this.

[dashpole]

I don't have any plans to implement exemplars for summary metrics, and I haven't personally heard any interest from users. I'm not opposed to the addition, as it might be useful in niche situations, but it does seem like more effort than it would be worth at this point.

[jsuereth]

Closing this as unplanned. We can reopen in the future if we feel summary metrics need exemplar support in OTLP.