-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubexit does not shutdown child when tombstone is written before it is started #8
Comments
RemcodM
added a commit
to RemcodM/kubexit
that referenced
this issue
Aug 10, 2020
This commit changes the `Watch` function inside the `tombstone` package to also emit an initial event besides the `fsnotify` events. This initial event is called immediatly when `Watch` is called and the watcher has been setup. This change allows kubexit to detect tombstones written before kubexit was started. This prevents possible race conditions as described by karlkfi#8. In order for this change to work, the `tombstone.EventHandler` type was changed. It now requires a function with 3 arguments: The graveyard, the tombstone and the operation instead of an `fsnotify.Event`. Reason being that the initial event is not an `fsnotify.Event`. The functions implementing an `tombstone.EventHandler` are changed accordingly. This change on its own introduces a new bug where the tombstone is written as part of an initial event, but the child process will still start because `child.Start()` is being called after the watcher has been setup. To overcome this issue, the shutdown state of the child is tracked in a new flag, which is set if `ShutdownNow()` or `ShutdownWithTimeout()` is executed.
RemcodM
added a commit
to RemcodM/kubexit
that referenced
this issue
Aug 10, 2020
This commit changes the `Watch` function inside the `tombstone` package to also emit an fake 'Created' event besides the real `fsnotify` events. This initial event is send out immediatly when `Watch` is called and the watcher has been setup. This change allows kubexit to detect tombstones written before kubexit was started. This prevents possible race conditions as described by karlkfi#8. This change on its own introduces a new bug where the tombstone is written as part of an initial event, but the child process will still start because `child.Start()` is being called after the watcher has been setup. To overcome this issue, the shutdown state of the child is tracked in a new flag, which is set if `ShutdownNow()` or `ShutdownWithTimeout()` is executed.
stephpalis
referenced
this issue
in acquia/fn-kubexit
Jul 13, 2021
This commit changes the `Watch` function inside the `tombstone` package to also emit an initial event besides the `fsnotify` events. This initial event is called immediatly when `Watch` is called and the watcher has been setup. This change allows kubexit to detect tombstones written before kubexit was started. This prevents possible race conditions as described by #8. In order for this change to work, the `tombstone.EventHandler` type was changed. It now requires a function with 3 arguments: The graveyard, the tombstone and the operation instead of an `fsnotify.Event`. Reason being that the initial event is not an `fsnotify.Event`. The functions implementing an `tombstone.EventHandler` are changed accordingly. This change on its own introduces a new bug where the tombstone is written as part of an initial event, but the child process will still start because `child.Start()` is being called after the watcher has been setup. To overcome this issue, the shutdown state of the child is tracked in a new flag, which is set if `ShutdownNow()` or `ShutdownWithTimeout()` is executed.
stephpalis
referenced
this issue
in acquia/fn-kubexit
Jul 13, 2021
* Ensure tombstones created before kubexit started are read This commit changes the `Watch` function inside the `tombstone` package to also emit an initial event besides the `fsnotify` events. This initial event is called immediatly when `Watch` is called and the watcher has been setup. This change allows kubexit to detect tombstones written before kubexit was started. This prevents possible race conditions as described by #8. In order for this change to work, the `tombstone.EventHandler` type was changed. It now requires a function with 3 arguments: The graveyard, the tombstone and the operation instead of an `fsnotify.Event`. Reason being that the initial event is not an `fsnotify.Event`. The functions implementing an `tombstone.EventHandler` are changed accordingly. This change on its own introduces a new bug where the tombstone is written as part of an initial event, but the child process will still start because `child.Start()` is being called after the watcher has been setup. To overcome this issue, the shutdown state of the child is tracked in a new flag, which is set if `ShutdownNow()` or `ShutdownWithTimeout()` is executed.
For future reader (and me), there are some working forks: |
11 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi again!
I'm back with a new race condition that we discovered while running jobs with kubexit on a very busy cluster. We have a job with 2 containers. Let's call them A and B. B is a sidecar for A. That means B is shutdown by kubexit whenever A exits. In the same way, A waits for B to be started before it starts up (birth dependency).
Most of the time, B is correctly shutdown. However, we have encoutered a case in which the cluster is very busy and the sidecar container is not executed directly. This causes A to timeout while waiting for B and a tombstone is written. However, after some time B is still started and this tombstone seems to be ignored. See the logs below.
Log for A:
Log for B:
Notice the timestamps: B is started at
08:53:45
, while A was already finished at08:53:14
. I think kubexit ignores existing tombstones when launching. When I have some spare time, I will look into it.The text was updated successfully, but these errors were encountered: