kubeadm: add docs for UpgradeAddonsBeforeControlPlane feature gate #41606

SataQiu · 2023-06-12T15:11:22Z

kubeadm: add docs for UpgradeAddonsBeforeControlPlane feature gate

Ref kubernetes/kubeadm#2346

neolit123

/sig cluster-lifecycle

some minor nits but, lgtm
thanks

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

neolit123 · 2023-06-12T15:20:05Z

wants to merge 1 commit into kubernetes:main from

should be for the dev-1.28 branch instead

netlify · 2023-06-12T15:52:37Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`9c3c6bf`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/64873b9c5c02130008c2689b
😎 Deploy Preview	https://deploy-preview-41606--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

neolit123 · 2023-06-12T16:04:24Z

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md

+   Before v1.27 (including v1.27), kubeadm will upgrade the addons (including CoreDNS and kube-proxy) immediately during 
+   `kubeadm upgrade apply`, regardless of whether there are other control plane instances that have not been upgraded, which 
+   may cause compatibility problems. Since v1.28, kubeadm will always checks whether all the control plane instances have 
+   been upgraded before starting to upgrade the addons. You **MUST** perform control plane nodes upgrade sequentially, and 


q: don't remember if i asked this. some users are known to not follow this rule today and they try to do parallel cp upgrades. will the logic we added in 1.28 support them or there are no guarantees - i.e. undefined behavior?

@neolit123 Because the check is only performed during kubeadm upgrade apply/node, when the control plane nodes are upgrading in parallel, there may be a race that all the kubeadm upgrade processes assume that there are unupgraded control plane instances, this will cause the addons to miss performing the upgrade. Complete serial execution is not required, but in order to ensure that the addons can be successfully upgraded during the last cp node upgrade, users should ensure that all the other cp nodes have been upgraded before doing the last cp node upgrade. In the future, we can add a new separate phase and call it after all the cp nodes have been upgraded, such as kubeadm upgrade addons. Then gradually deprecate upgrading addons during the upgrade apply/node phase. But this will change the upgrade workflow and may take a long time.

If parallel cp upgrade is necessary, the user needs to make additional checks to see if the addons have been successfully upgraded or always to call kubeadm upgrade node or kubeadm upgrade addons(in the future) on the last cp node again to ensure that the addons can be upgraded successfully. This should be safe because the upgrade operation is idempotent.

thanks for the clarification.

sftim

Hi. Here's how I'd change this - what do you think @SataQiu ?

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md

SataQiu · 2023-06-13T10:59:49Z

thanks @sftim @neolit123 PTAL again!

neolit123

/lgtm

k8s-ci-robot · 2023-06-13T12:11:57Z

LGTM label has been added.

Git tree hash: 2829a9f7314fa1cd5e424518e595d7a4fd6c46a6

SataQiu · 2023-06-17T15:22:12Z

hi @tengqm
Would you mind take a look?

tengqm · 2023-06-18T01:56:17Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

@@ -160,6 +160,7 @@ Feature | Default | Alpha | Beta | GA
 `PublicKeysECDSA` | `false` | 1.19 | - | -
 `RootlessControlPlane` | `false` | 1.22 | - | -
 `UnversionedKubeletConfigMap` | `true` | 1.22 | 1.23 | 1.25
+`UpgradeAddonsBeforeControlPlane` | `false` | - | - | -


Is this alpha, or beta?

Thanks @tengqm
Sorry for the late reply.

It isn't a new feature, it's more like a bugfix. So this feature gate has been marked deprecated since it was created.
So it's not in Alpha stage, and will never be Beta or GA.
For more context: kubernetes/kubernetes#116570 (comment)

In that case, we should document this in the second table (gates for GA'ed or deprecated feature).

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

tengqm · 2023-06-18T02:00:43Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

+instances upgrade sequentially or at least ensure that the last control plane instance is upgraded after all the other control
+plane instances have been upgraded, and the addons upgrade will be performed after the last control plane instance is upgraded.


Ensure "that the last control plane instance is upgraded after all the other control
plane instances have been upgraded" ...

This is so verbose. Does this mean that we need to ensure all control plane instances are properly upgraded?

This is because the control plane node upgrade needs to ensure the sequence. In a HA cluster, the upgrade of the last control plane node cannot be performed until all the other control plane nodes have been upgraded successfully.

tengqm · 2023-06-18T02:02:37Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

+has been upgraded by checking whether the image of the kube-apiserver Pod has been upgraded. You **MUST** perform control plane
+instances upgrade sequentially or at least ensure that the last control plane instance is upgraded after all the other control
+plane instances have been upgraded, and the addons upgrade will be performed after the last control plane instance is upgraded.
+The deprecated `UpgradeAddonsBeforeControlPlane` feature gate gives you a chance to keep the old upgrade behavior. You should not


Is the gate defaulted to true or false?
It is introduced as a deprecated feature, right?

Yes, the deprecated feature gate is defaulted to false.

tengqm · 2023-06-18T02:03:34Z

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md

+instances upgrade sequentially or at least ensure that the last control plane instance is upgraded after all the other control
+plane instances have been upgraded, and the addons upgrade will be performed after the last control plane instance is upgraded.
+The deprecated `UpgradeAddonsBeforeControlPlane` feature gate gives you a chance to keep the old upgrade behavior. You should not
+need this old behavior; if you do, you should change your cluster or upgrade processes but please do not rely on it, as this


"please do not rely on it" ...
What does this "it" refer to?

"it" means the UpgradeAddonsBeforeControlPlane feature gate.

"please do not rely on it" has been removed now

tengqm · 2023-06-21T22:38:59Z

/approve

k8s-ci-robot · 2023-06-21T22:39:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tengqm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/OWNERS~~ [tengqm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

neolit123

/lgtm

k8s-ci-robot · 2023-06-22T14:58:21Z

LGTM label has been added.

Git tree hash: 1c39610de7f763532d027aefd8c517204bb076d6

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 12, 2023

k8s-ci-robot requested review from mehabhalodiya and nate-double-u June 12, 2023 15:11

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jun 12, 2023

SataQiu requested review from neolit123 and removed request for nate-double-u and mehabhalodiya June 12, 2023 15:11

k8s-ci-robot added language/en Issues or PRs related to English language sig/docs Categorizes an issue or PR as relevant to SIG Docs. labels Jun 12, 2023

SataQiu requested a review from tengqm June 12, 2023 15:11

neolit123 reviewed Jun 12, 2023

View reviewed changes

k8s-ci-robot added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label Jun 12, 2023

k8s-ci-robot added the language/ko Issues or PRs related to Korean language label Jun 12, 2023

SataQiu changed the base branch from main to dev-1.18 June 12, 2023 15:23

SataQiu changed the base branch from dev-1.18 to main June 12, 2023 15:23

SataQiu force-pushed the add-docs-for-UpgradeAddonsBeforeControlPlane branch from 54a51ec to 9c3c6bf Compare June 12, 2023 15:36

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 12, 2023

SataQiu changed the base branch from main to dev-1.28 June 12, 2023 15:37

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 12, 2023

SataQiu mentioned this pull request Jun 12, 2023

mutable upgrades: upgrade the kube-proxy DS after all control-plane components kubernetes/kubeadm#2346

Closed

4 tasks

SataQiu removed the language/ko Issues or PRs related to Korean language label Jun 12, 2023

neolit123 reviewed Jun 12, 2023

View reviewed changes

content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade.md Outdated Show resolved Hide resolved

neolit123 reviewed Jun 12, 2023

View reviewed changes

content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md Outdated Show resolved Hide resolved

neolit123 reviewed Jun 12, 2023

View reviewed changes

sftim reviewed Jun 12, 2023

View reviewed changes

SataQiu force-pushed the add-docs-for-UpgradeAddonsBeforeControlPlane branch from 9c3c6bf to c43c101 Compare June 13, 2023 10:57

SataQiu force-pushed the add-docs-for-UpgradeAddonsBeforeControlPlane branch from c43c101 to fe98320 Compare June 13, 2023 11:03

neolit123 reviewed Jun 13, 2023

View reviewed changes

k8s-ci-robot assigned neolit123 Jun 13, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 13, 2023

SataQiu requested a review from sftim June 14, 2023 14:40

tengqm reviewed Jun 18, 2023

View reviewed changes

kubeadm: add docs for UpgradeAddonsBeforeControlPlane feature gate

d5a2a69

SataQiu force-pushed the add-docs-for-UpgradeAddonsBeforeControlPlane branch from fe98320 to d5a2a69 Compare June 21, 2023 05:35

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 21, 2023

k8s-ci-robot requested a review from neolit123 June 21, 2023 05:35

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 21, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 21, 2023

neolit123 reviewed Jun 22, 2023

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 22, 2023

k8s-ci-robot merged commit 30f8c53 into kubernetes:dev-1.28 Jun 22, 2023

k8s-ci-robot added this to the 1.28 milestone Jun 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: add docs for UpgradeAddonsBeforeControlPlane feature gate #41606

kubeadm: add docs for UpgradeAddonsBeforeControlPlane feature gate #41606

SataQiu commented Jun 12, 2023

neolit123 left a comment

neolit123 commented Jun 12, 2023

netlify bot commented Jun 12, 2023

neolit123 Jun 12, 2023

SataQiu Jun 13, 2023 •

edited

Loading

SataQiu Jun 13, 2023 •

edited

Loading

neolit123 Jun 13, 2023

sftim left a comment

SataQiu commented Jun 13, 2023

neolit123 left a comment

k8s-ci-robot commented Jun 13, 2023

SataQiu commented Jun 17, 2023

tengqm Jun 18, 2023

SataQiu Jun 21, 2023

tengqm Jun 21, 2023

SataQiu Jun 21, 2023 •

edited

Loading

tengqm Jun 18, 2023

SataQiu Jun 21, 2023

tengqm Jun 18, 2023

SataQiu Jun 21, 2023

tengqm Jun 18, 2023

SataQiu Jun 21, 2023

SataQiu Jun 21, 2023

tengqm commented Jun 21, 2023

k8s-ci-robot commented Jun 21, 2023

neolit123 left a comment

k8s-ci-robot commented Jun 22, 2023

		instances upgrade sequentially or at least ensure that the last control plane instance is upgraded after all the other control
		plane instances have been upgraded, and the addons upgrade will be performed after the last control plane instance is upgraded.

kubeadm: add docs for UpgradeAddonsBeforeControlPlane feature gate #41606

kubeadm: add docs for UpgradeAddonsBeforeControlPlane feature gate #41606

Conversation

SataQiu commented Jun 12, 2023

neolit123 left a comment

Choose a reason for hiding this comment

neolit123 commented Jun 12, 2023

netlify bot commented Jun 12, 2023

✅ Pull request preview available for checking

Choose a reason for hiding this comment

SataQiu Jun 13, 2023 • edited Loading

Choose a reason for hiding this comment

SataQiu Jun 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim left a comment

Choose a reason for hiding this comment

SataQiu commented Jun 13, 2023

neolit123 left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 13, 2023

SataQiu commented Jun 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SataQiu Jun 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tengqm commented Jun 21, 2023

k8s-ci-robot commented Jun 21, 2023

neolit123 left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jun 22, 2023

SataQiu Jun 13, 2023 •

edited

Loading

SataQiu Jun 13, 2023 •

edited

Loading

SataQiu Jun 21, 2023 •

edited

Loading