Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Informer.watch is not guaranteed to receive all changes during restart #599

Open
DocX opened this issue Jan 5, 2023 · 0 comments
Open

Comments

@DocX
Copy link
Contributor

DocX commented Jan 5, 2023

Using the Infomer.watch method that yields notices from the watch endpoint is not guaranteed to receive all events in case the informer restarts the watcher:

  1. Informer fills/replaces cache from get request and stores the resource version
  2. Informer starts watch request with the resource version from the get
  3. Watch request stops (timeout expires, connection drops, ...)
  4. Some changes happen in Kubernetes cluster before the next get request (those will not be yielded to the watch block)
  5. Repeat 1: the get will receive the state with the changes happened in 4 and update list
  6. Repeat 2: but the watch now will start after those changes, so watch will not be yielded with the changes happened in 4.

Potential solutions:

  1. Document the behaviour of that watch interface can miss changes (and that list is the only source of truth)
  2. Only request get once when informer.start_worker is called, when watch request stops, only restart the watch with the latest seen resource version
@DocX DocX changed the title Informer.watch is not guaranteed to receive all changes Informer.watch is not guaranteed to receive all changes during restart Jan 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant