Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus - Exemplar label name is invalid when filtering attributes in a custom view #5936

Open
StarpTech opened this issue Oct 30, 2024 · 4 comments · May be fixed by #5995
Open

Prometheus - Exemplar label name is invalid when filtering attributes in a custom view #5936

StarpTech opened this issue Oct 30, 2024 · 4 comments · May be fixed by #5995
Assignees
Labels
area:metrics Part of OpenTelemetry Metrics bug Something isn't working pkg:exporter:prometheus Related to the Prometheus exporter package
Milestone

Comments

@StarpTech
Copy link

StarpTech commented Oct 30, 2024

Description

After upgrading from go.opentelemetry.io/otel/exporters/prometheus v0.50.0 to go.opentelemetry.io/otel/exporters/prometheus v0.53.0 I see the following error exemplar label name \"wg.operation.hash\" is invalid.

We use a custom view to filter metric attributes with high cardinality. While the metric is definitely dropped in the view it seems the data is still exported on the prometheus exporter.

It is necessary to checkout 664a075 to not run into a different exemplar prometheus issue.

var opts []sdkmetric.Option

	// Exclude attributes from metrics

	attributeFilter := func(value attribute.KeyValue) bool {
		if isKeyInSlice(value.Key, defaultExcludedOtelKeys) {
			return false
		}
		name := sanitizeName(string(value.Key))
		for _, re := range c.Prometheus.ExcludeMetricLabels {
			if re.MatchString(name) {
				return false
			}
		}
		return true
	}

	msBucketHistogram := sdkmetric.AggregationExplicitBucketHistogram{
		Boundaries: msBucketsBounds,
	}
	bytesBucketHistogram := sdkmetric.AggregationExplicitBucketHistogram{
		Boundaries: bytesBucketBounds,
	}

	var view sdkmetric.View = func(i sdkmetric.Instrument) (sdkmetric.Stream, bool) {
		// In a custom View function, we need to explicitly copy the name, description, and unit.
		s := sdkmetric.Stream{Name: i.Name, Description: i.Description, Unit: i.Unit}

		// Use different histogram buckets for PrometheusConfig
		if i.Unit == unitBytes && i.Kind == sdkmetric.InstrumentKindHistogram {
			s.Aggregation = bytesBucketHistogram
		} else if i.Unit == unitMilliseconds && i.Kind == sdkmetric.InstrumentKindHistogram {
			s.Aggregation = msBucketHistogram
		}

		// Filter out metrics that match the excludeMetrics regexes
		for _, re := range c.Prometheus.ExcludeMetrics {
			promName := sanitizeName(i.Name)
			if re.MatchString(promName) {
				// Drop the metric
				s.Aggregation = sdkmetric.AggregationDrop{}
				return s, true
			}
		}

		// Filter out attributes that match the excludeMetricAttributes regexes
		s.AttributeFilter = attributeFilter

		return s, true
	}

	opts = append(opts, sdkmetric.WithView(view))

Stacktrace of the OTEL error

16:34:58 PM ERROR trace/meter.go:239 otel error {"hostname": "dustins-MacBook-Pro.local", "pid": 79745, "component": "@wundergraph/router", "service_version": "dev", "error": "exemplar label name \"wg.operation.hash\" is invalid"}
github.com/wundergraph/cosmo/router/pkg/trace.NewTracerProvider.func3
        /Users/starptech/p/wundergraph/cosmo/router/pkg/trace/meter.go:239
go.opentelemetry.io/otel.ErrorHandlerFunc.Handle
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/[email protected]/error_handler.go:26
go.opentelemetry.io/otel.Handle
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/[email protected]/handler.go:33
go.opentelemetry.io/otel/exporters/prometheus.addExemplars[...]
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/[email protected]/exporter.go:535
go.opentelemetry.io/otel/exporters/prometheus.addHistogramMetric[...]
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/[email protected]/exporter.go:260
go.opentelemetry.io/otel/exporters/prometheus.(*collector).Collect
        /Users/starptech/go/pkg/mod/go.opentelemetry.io/otel/exporters/[email protected]/exporter.go:229
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1
        /Users/starptech/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:458

Environment

  • OS: iOS
  • Architecture: ARM
  • Go Version: 1.21
  • opentelemetry-go version: 1.31.1

Expected behavior

It should be possible to filter attributes like before including prometheus exporter support

@StarpTech StarpTech added the bug Something isn't working label Oct 30, 2024
@StarpTech
Copy link
Author

Workaround. Disabling exemplars for all

sdkmetric.WithExemplarFilter(exemplar.AlwaysOffFilter)

@pellared pellared added area:metrics Part of OpenTelemetry Metrics pkg:exporter:prometheus Related to the Prometheus exporter package labels Nov 21, 2024
@pellared pellared removed this from Logs SIG Nov 21, 2024
@github-project-automation github-project-automation bot moved this to Needs triage in Go: Triage Nov 21, 2024
@MrAlias
Copy link
Contributor

MrAlias commented Nov 21, 2024

cc @dashpole

@MrAlias MrAlias moved this from Needs triage to High priority in Go: Triage Nov 21, 2024
@MrAlias MrAlias added this to the v1.33.0 milestone Nov 21, 2024
@dashpole dashpole self-assigned this Nov 21, 2024
@dashpole
Copy link
Contributor

Related: prometheus/prometheus#15259

To fix this, we need to sanitize exemplar keys here:

labels[string(attr.Key)] = attr.Value.Emit()

We should use the same approach used for label keys:

key := model.EscapeName(string(kv.Key), model.NameEscapingScheme)

@dashpole dashpole linked a pull request Nov 21, 2024 that will close this issue
@dashpole
Copy link
Contributor

Fix out: #5995. @StarpTech are you able to verify that it fixes your issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:metrics Part of OpenTelemetry Metrics bug Something isn't working pkg:exporter:prometheus Related to the Prometheus exporter package
Projects
Status: High priority
Development

Successfully merging a pull request may close this issue.

4 participants