Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

QFE joining ring post upgrading to 2.14.1 causing gossip alerts to misfire #9841

Open
rishabhkumar92 opened this issue Nov 5, 2024 · 9 comments · May be fixed by #10289
Open

QFE joining ring post upgrading to 2.14.1 causing gossip alerts to misfire #9841

rishabhkumar92 opened this issue Nov 5, 2024 · 9 comments · May be fixed by #10289
Labels
bug Something isn't working

Comments

@rishabhkumar92
Copy link

Describe the bug

QFE joining ring post upgrading to 2.14.1 which is causing gossip alerts (ref) to misfire

To Reproduce

Steps to reproduce the behavior:

  1. Start Mimir 2.14.1
  2. Noticed alert firing and confirmed that QFE has joined the ring

Expected behavior

QFE shouldn't join the ring and this behavior was working as expected till 2.13

Environment

  • Infrastructure: Kubernetes

Additional Context

NA
@rishabhkumar92
Copy link
Author

Image

@armandgrillet armandgrillet added the bug Something isn't working label Nov 8, 2024
@rishabhkumar92
Copy link
Author

Any update on this one? cc: @armandgrillet

@AelwynBo
Copy link

Hello,
I am facing the same issue in a setup with virtual machines since 2.14.0 upgrade.

@fgouteroux
Copy link
Contributor

any news on this guys ?

@francoposa francoposa linked a pull request Dec 19, 2024 that will close this issue
Draft
4 tasks
@fgouteroux
Copy link
Contributor

Before the version 2.14, when we make a request in /memberlist we got the message This instance doesn't use memberlist..

Something has change as now the queryfrontend is now a member in Memberlist Cluster. So the alert should be adapted like purposed in PR #10289

@pracucci
Copy link
Collaborator

any news on this guys ?

It should be a side-effect of the query-frontend joining memberlist because the ring is used in the query-frontend when the new experimental ingest storage is enabled. Due to how internal modules are initialised in Mimir, memberlist will be initialised even if the ring will not be effectively used.

@fgouteroux
Copy link
Contributor

any news on this guys ?

It should be a side-effect of the query-frontend joining memberlist because the ring is used in the query-frontend when the new experimental ingest storage is enabled. Due to how internal modules are initialised in Mimir, memberlist will be initialised even if the ring will not be effectively used.

Does this behavior will change in next release ? For me it's ok if the query-frontend join memberlist, we know are many nodes we have in the cluster.

@pracucci
Copy link
Collaborator

any news on this guys ?

It should be a side-effect of the query-frontend joining memberlist because the ring is used in the query-frontend when the new experimental ingest storage is enabled. Due to how internal modules are initialised in Mimir, memberlist will be initialised even if the ring will not be effectively used.

Does this behavior will change in next release ? For me it's ok if the query-frontend join memberlist, we know are many nodes we have in the cluster.

I don't expect it to change.

@fgouteroux
Copy link
Contributor

@pracucci thx for the explanation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix MimirGossipMembersTooLow alert mixin: add query-frontend to ring members grafana/mimir
5 participants
@fgouteroux @pracucci @armandgrillet @rishabhkumar92 @AelwynBo