Prometheus - Exemplar label name is invalid when filtering attributes in a custom view

[StarpTech]

Description

After upgrading from go.opentelemetry.io/otel/exporters/prometheus v0.50.0 to go.opentelemetry.io/otel/exporters/prometheus v0.53.0 I see the following error exemplar label name \"wg.operation.hash\" is invalid.

We use a custom view to filter metric attributes with high cardinality. While the metric is definitely dropped in the view it seems the data is still exported on the prometheus exporter.

It is necessary to checkout 664a075 to not run into a different exemplar prometheus issue.

var opts []sdkmetric.Option

	// Exclude attributes from metrics

	attributeFilter := func(value attribute.KeyValue) bool {
		if isKeyInSlice(value.Key, defaultExcludedOtelKeys) {
			return false
		}
		name := sanitizeName(string(value.Key))
		for _, re := range c.Prometheus.ExcludeMetricLabels {
			if re.MatchString(name) {
				return false
			}
		}
		return true
	}

	msBucketHistogram := sdkmetric.AggregationExplicitBucketHistogram{
		Boundaries: msBucketsBounds,
	}
	bytesBucketHistogram := sdkmetric.AggregationExplicitBucketHistogram{
		Boundaries: bytesBucketBounds,
	}

	var view sdkmetric.View = func(i sdkmetric.Instrument) (sdkmetric.Stream, bool) {
		// In a custom View function, we need to explicitly copy the name, description, and unit.
		s := sdkmetric.Stream{Name: i.Name, Description: i.Description, Unit: i.Unit}

		// Use different histogram buckets for PrometheusConfig
		if i.Unit == unitBytes && i.Kind == sdkmetric.InstrumentKindHistogram {
			s.Aggregation = bytesBucketHistogram
		} else if i.Unit == unitMilliseconds && i.Kind == sdkmetric.InstrumentKindHistogram {
			s.Aggregation = msBucketHistogram
		}

		// Filter out metrics that match the excludeMetrics regexes
		for _, re := range c.Prometheus.ExcludeMetrics {
			promName := sanitizeName(i.Name)
			if re.MatchString(promName) {
				// Drop the metric
				s.Aggregation = sdkmetric.AggregationDrop{}
				return s, true
			}
		}

		// Filter out attributes that match the excludeMetricAttributes regexes
		s.AttributeFilter = attributeFilter

		return s, true
	}

	opts = append(opts, sdkmetric.WithView(view))

Stacktrace of the OTEL error

16:34:58 PM ERROR trace/meter.go:239 otel error {"hostname": "dustins-MacBook-Pro.local", "pid": 79745, "component": "@wundergraph/router", "service_version": "dev", "error": "exemplar label name \"wg.operation.hash\" is invalid"}
github.com/wundergraph/cosmo/router/pkg/trace.NewTracerProvider.func3
        /Users/starptech/p/wundergraph/cosmo/router/pkg/trace/meter.go:239
go.opentelemetry.io/otel.ErrorHandlerFunc.Handle
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/[email protected]/error_handler.go:26
go.opentelemetry.io/otel.Handle
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/[email protected]/handler.go:33
go.opentelemetry.io/otel/exporters/prometheus.addExemplars[...]
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/[email protected]/exporter.go:535
go.opentelemetry.io/otel/exporters/prometheus.addHistogramMetric[...]
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/[email protected]/exporter.go:260
go.opentelemetry.io/otel/exporters/prometheus.(*collector).Collect
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/[email protected]/exporter.go:229
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        /Users/starptech/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:458

Environment

• OS: iOS
• Architecture: ARM
• Go Version: 1.21
• opentelemetry-go version: 1.31.1

Expected behavior

It should be possible to filter attributes like before including prometheus exporter support

[StarpTech]

Workaround. Disabling exemplars for all

sdkmetric.WithExemplarFilter(exemplar.AlwaysOffFilter)

[MrAlias]

cc @dashpole

[dashpole]

Related: prometheus/prometheus#15259

To fix this, we need to sanitize exemplar keys here:

opentelemetry-go/exporters/prometheus/exporter.go

Line 550 in 99c3c66

labels[string(attr.Key)] = attr.Value.Emit()

We should use the same approach used for label keys:

opentelemetry-go/exporters/prometheus/exporter.go

Line 335 in 99c3c66

key := model.EscapeName(string(kv.Key), model.NameEscapingScheme)

[dashpole]

Fix out: #5995. @StarpTech are you able to verify that it fixes your issue?