Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Prometheus metrics plugin will throw Incorrect number of labels when partitioned table and non-partitioned table mixed in cluster #302

Open
1 of 2 tasks
luoyuxia opened this issue Jan 2, 2025 · 0 comments
Labels
bug Something isn't working component=metrics

Comments

@luoyuxia
Copy link
Collaborator

luoyuxia commented Jan 2, 2025

Search before asking

  • I searched in the issues and found nothing similar.

Fluss version

main

Minimal reproduce step

1: enable prometheus metric reporter
2: create a partitioned table and non-partitioned table
3: then the following waning message will be thrown

2024-12-27 16:48:12,199 WARN  com.alibaba.fluss.metrics.registry.MetricRegistryImpl        [] - Error while handling metric: messagesInPerSecond.
java.lang.IllegalArgumentException: Incorrect number of labels.
	at io.prometheus.client.SimpleCollector.setChild(SimpleCollector.java:136) ~[?:?]
	at com.alibaba.fluss.metrics.prometheus.PrometheusReporter.addMetric(PrometheusReporter.java:214) ~[?:?]
	at com.alibaba.fluss.metrics.prometheus.PrometheusReporter.notifyOfAddedMetric(PrometheusReporter.java:153) ~[?:?]
	at com.alibaba.fluss.metrics.registry.MetricRegistryImpl.forAllReporters(MetricRegistryImpl.java:267) ~[fluss-server-0.6-SNAPSHOT.jar:0.6-SNAPSHOT]
	at com.alibaba.fluss.metrics.registry.MetricRegistryImpl.register(MetricRegistryImpl.java:214) ~[fluss-server-0.6-SNAPSHOT.jar:0.6-SNAPSHOT]
	at com.alibaba.fluss.metrics.groups.AbstractMetricGroup.addMetric(AbstractMetricGroup.java:292) ~[fluss-server-0.6-SNAPSHOT.jar:0.6-SNAPSHOT]
	at com.alibaba.fluss.metrics.groups.AbstractMetricGroup.meter(AbstractMetricGroup.java:250) ~[fluss-server-0.6-SNAPSHOT.jar:0.6-SNAPSHOT]

What doesn't meet your expectations?

Should print warn message

Anything else?

1: for partitioned table and metrics name messagesInPerSecond, assuming m1, it'll be of scope name fluss_tabletserver_table__messagesInPerSecond with labels {database=test_db_1, cluster_id=fluss, partition=2027, host=127.0.0.1, tablet_type=kv, server_id=2, table=test_pk_table_1}. Note the label count is 7

2: for non-partitioned table and metrics name fluss_tabletserver_table__messagesInPerSecond, assuming m2, it'll be of scope name
fluss_tabletserver_table__messagesInPerSecond with labels {database=test_db_11, cluster_id=fluss, host=127.0.0.1, tablet_type=kv, server_id=2, table=tt}. Note the label count is 6

First, it registers m1 with a new created Collector c1 to Prometheus reporter. Then when add m2, it has same scope name with m1, so it use c1 to add. However, c1 is expected to with label count 7, m2 is with label count 6. So, Incorrect number of labels. is thrown..

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@luoyuxia luoyuxia added bug Something isn't working component=metrics labels Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working component=metrics
Projects
None yet
Development

No branches or pull requests

1 participant