Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider maintenance window when migrating master pods #2810

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

hughcapet
Copy link
Member

Extend MaintenanceWindows parameter usage

@hughcapet hughcapet added the major Major feature changes or updates, e.g. feature rollout to a new country, new API calls. label Dec 4, 2024
@hughcapet hughcapet marked this pull request as ready for review December 23, 2024 12:09
pkg/cluster/util.go Outdated Show resolved Hide resolved
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from 7fade35 to 5cab328 Compare January 3, 2025 16:39
@idanovinda
Copy link
Member

👍

@FxKu
Copy link
Member

FxKu commented Jan 6, 2025

If I recall correctly moving the primary is a once or never operation. Removing the node readiness label triggers an node event which the operator is listening on to perform the migration. If the migration does not work the operator will not try again, because no other node event is send. Hence the primary stays on the node and blocks further operators by the K8s admins. We have seen it in the past where we were asked to trigger a switchover to unblock K8s updates.

Therefore, I think we greatly increase the possibility that pod migrations will not happen at all if they are omitted due to maintenance window restrictions. However, we could instead mark the primary pod to be rotated so another sync will trigger a switchover so that the old instance can be removed by K8s drainage.

@hughcapet hughcapet marked this pull request as draft January 7, 2025 08:14
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from 04a488a to f488bee Compare January 8, 2025 07:56
@hughcapet hughcapet force-pushed the maintwindow-master-migration branch from f488bee to 9689f25 Compare January 8, 2025 13:36
@hughcapet hughcapet marked this pull request as ready for review January 8, 2025 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
major Major feature changes or updates, e.g. feature rollout to a new country, new API calls.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants