Is it possible to make the error retrieving resource lock for leader election Fatal?

[smg247]

We have recently uncovered an error in our boskos instance like the following:

E0710 21:02:45.211259 1 leaderelection.go:330] error retrieving resource lock ci/boskos-cleaner-leaderlock: leases.coordination.k8s.io "boskos-cleaner-leaderlock" is forbidden: User "system:serviceaccount:ci:boskos" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ci"

This went unnoticed for quite some time as it is not a fatal error so the pod wasn't killed (and our alerting was never triggered). Is it possible to make the following errors Fatal?

• https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L332
• https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L336

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Gatete-Bruno]

E0611 21:13:40.731888 1 leaderelection.go:332] error retrieving resource lock email-operator-system/650681eb.batman.example.com: leases.coordination.k8s.io "650681eb.batman.example.com" is forbidden: User "system:serviceaccount:email-operator-system:email-operator-email-operator-controller-manager" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "email-operator-system"

[maggie44]

I am running in to this issue too. Three pods were all failing as they were unable to access the lease, and yet the pods remained healthy so the issue went unnoticed. It led to a lot of issues.

Happy to raise a PR if we can come up with a solution. I had a look through the code with the hope of finding a non-breaking solution. Exiting with an error could potentially be disruptive to some as current behaviour is during temporary outages there would be a retry.

Another option could be adding in a check as part of the HealthzAdaptor:

client-go/tools/leaderelection/healthzadaptor.go

Line 45 in 4609339

func (l *HealthzAdaptor) Check(req *http.Request) error {

There isn't an obvious interface for checking for these errors though, or for enacting a ping to the API.

[maggie44]

/reopen

[k8s-ci-robot]

@maggie44: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[smg247]

/reopen

[k8s-ci-robot]

@smg247: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.