How to deal with removal of v1beta1 CRD removals in Kubernetes 1.22 / OpenShift 4.9

[dmesser]

📢 Kubernetes 1.22 API deprecations and OpenShift 4.9

What is the big deal? Kubernetes 1.22 removes some long deprecated APIs, among the CustomResourceDefinitions in the version v1beta1. OpenShift 4.9 is based upon Kubernetes 1.22 and consequently will not allow to use this API any more. What this means for contributors

• If your operator uses v1beta1 CRDs it will not work on OpenShift 4.9+, the installation will be blocked
• Users of OpenShift 4.8 trying to update to 4.9 will be warned about updating their cluster if they have such an operator installed
• to make your operator compatible with Kubernetes 1.22 and OpenShift 4.9, simply migrate your CRDs to version v1, OLM will handle the upgrade for you.
• if your operator uses webhooks, OLM will handle the creation / migration of your webhooks to version v1

👍 Selecting the right OpenShift version for your Operator

The OLM packaging format allows to specify with which OpenShift releases your operator is compatible and to which OpenShift versions it should be published:

• in the ClusterServiceVersion you can add the annotation olm.properties: '[{"type": "olm.maxOpenShiftVersion", "value": "4.8"}]' in metadata.annotations to specify the highest OpenShift version your operator is compatible with
• in the annotations.yaml of the bundle format you can specify the label com.redhat.openshift.versions: "v4.6-v4.8" which will instruct the publishing pipeline of this repository to only publish the operator version in the catalogs of the OpenShift version 4.6, 4.7 and 4.8

If your operator isn't ready to move to v1 CRDs or remove usage of any other deprecated APIs that will be removed in Kubernetes 1.22 / OpenShift 4.9, please use the above metadata tuneables when publishing your Operators.

The maintainers of the OpenShift community operators catalog already updated all existing operators in this pull request.

If your operator still uses the packagemanifest you cannot select in which OpenShift versions your operator will appear. Please consider updating to the bundle format using the documentation.

⚙️ Pipeline changes

The validation and publishing pipeline of this repository will adapt the following changes before October 2021:

• new submissions in bundle format will raise warnings if they contain deprecated APIs and are configured to also publish to OpenShift 4.9 and newer, later they will be completely blocked from merging
• new submissions in packagemanifest format will get auto-converted to bundle format in the background and will be restricted from publishing to 4.9 if they use deprecated APIs, auto-merging such PRs will be stopped until the contributors acknowledges this behavior

🚀 Get started updating your operator

If you are unsure if your operator is impacted by those API removals in Kubernetes 1.22 / OpenShift 4.9, you can use the operator-sdk to run a validation suite:

operator-sdk bundle validate ./bundle --select-optional suite=operatorframework --optional-values=k8s-version=1.22

If the test results show that the operator uses deprecated APIs, refer to the Kubernetes documentation to find out what changed in the newer versions and how to migrate.

[camilamacedo86]

💡 How to make your project compatible with k8s 1.22 and OCP 4.9?

Due to the number of options available to build Operators, it is hard to provide direct guidance on updating your operator to support Kubernetes 1.22. Recent versions of the OperatorSDK greater than 1.0.0 and Kubebuilder greater than 3.0.0 scaffold your project with the latest versions of these APIs (all that is generated by tools only). See the guides to upgrade your projects with OperatorSDK Golang, Ansible, Helm or the Kubebuilder one. For APIs other than the ones mentioned above, you will have to check your code for usage of removed API versions and upgrade to newer APIs. The details of this depend on your codebase.

Following a collection of how to's and tips to help you find if your project is using these removed APIs, how to update them, and how to test your changes. We hope that helps you out.

🔥 Checking the usage

• By looking in the code source:

You can grep your project's code to looking for all usage of these APIs. All APIs removed on k8s 1.22 / OCP 4.9 can be found in Removed APIs by release - 1.22, e.g:

$ grep -rni apiextensions.k8s.io/v1beta1 *
bundle/manifests/cache.example.com_memcacheds.yaml:1:apiVersion: apiextensions.k8s.io/v1beta1
config/crd/patches/cainjection_in_memcacheds.yaml:3:apiVersion: apiextensions.k8s.io/v1beta1
config/crd/patches/webhook_in_memcacheds.yaml:3:apiVersion: apiextensions.k8s.io/v1beta1
config/crd/bases/cache.example.com_memcacheds.yaml:3:apiVersion: apiextensions.k8s.io/v1beta1

• By using OperatorSDK(latest versions only) to validate the manifest in your bundle:

$  operator-sdk bundle validate <your bundle path> --select-optional suite=operatorframework --optional-values=k8s-version=1.22

$  operator-sdk bundle validate ./bundle --select-optional suite=operatorframework --optional-values=k8s-version=1.22
... 
ERRO[0000] Error: Value memcached-operator.v0.0.1: this bundle uses APIs that were deprecated and removed in v1.22. More info: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-22. Migrate the API(s) for CRD: (["memcacheds.cache.example.com"]) 

NOTE This check is only able to look at the manifests shipped inside of the bundle. Suppose your operator uses the removed APIs, but they are not shipped on the bundle, OR they depend on a project that uses them and will not work on OCP 4.9+. In that case, unfortunately, we can not identify these scenarios. You will need to determine if this is your case by code review and QE testing on the clusters.

• By checking the events/alerts

If you deploy your Operator on the cluster and perform regression tests, the APIs will raise alerts/events to notify you about the usage of APIs that will be removed in the next versions:

$ kubectl get events or $ oc get events

More information on this is available in this blog post.

NOTE: Events are also recorded in the Kubernetes API audit logs. Once the logs have been downloaded, deprecation events can easily be searched with:

$ grep "removed-release.*1.22" audit*.log

🔨 Updating your project

Now, you need to ensure that your project will no longer use these removed APIs on 1.22/OCP 4.9 and start to use the latest version(s).

For the k8s manifests, you need to re-generate them using the new versions. However, you might be importing them into your codebase. Then, you ought to replace the imports, e.g.: replace theapiused/v1beta1 with its latest version theapiused/v1 as well.

In the most common scenarios, projects are affected by using the k8s API apiextensions.k8s.io/v1beta1 for CRDs. In this way, follows an example and tips to migrate your CRDs from v1beta1 to (v1 format).

Tips and example(s) over how to migrate only CRDs

If you are using OperatorSDK CLI tool version >= v0.18.x < 1.0.0 you can:

$ operator-sdk generate crds --crd-version=v1
INFO[0000] Running CRD generator.                       
INFO[0000] CRD generation complete.

If you have been generating the manifests with controller-gen or manually then, you can use its upper version >= v0.4.1 and re-generate them:

$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role  paths="./..." output:crd:artifacts:config=<add here the output-path>

OR

$ controller-gen crd:crdVersions=v1,preserveUnknownFields=false rbac:roleName=manager-role paths="./..." output:crd:artifacts:config=<add here the output-path>

Here’s a full example:

$ go get sigs.k8s.io/controller-tools/cmd/[email protected]
$ controller-gen --version
Version: v0.4.1
$ controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role paths="./..." output:crd:artifacts:config=deploy/crds/

NOTE To understand the above option preserveUnknownFields=false see this k8s doc.

See that in a live cluster, you can invoke Kubernetes’ conversion functionality by applying a crd.v1beta1 and then kubectl get a crd.v1 to view its converted format.

Now, you have your CRD using the latest version to be shipped on the new bundle version.

NOTE: You are probably not using OperatorSDK CLI >= 1.0.0 because projects migrated to versions >= 1.0.0 are scaffolded by default using the latest K8S API versions. However, by using the latest scaffold you can create and update your bundle via $ make bundle. Also, see FAQ-Can I customize the projects initialized with operator-sdk?.

If your project uses Webhooks:

Add the makers sideEffects=none and admissionReviewVersions to your webhook (e.g. memcached-operator/api/v1alpha1/memcached_webhook.go):

Example:

//+kubebuilder:webhook:path=/mutate-cache-example-com-v1alpha1-memcached,mutating=true,failurePolicy=fail,sideEffects=None,groups=cache.example.com,resources=memcacheds,verbs=create;update,versions=v1alpha1,name=mmemcached.kb.io,admissionReviewVersions={v1,v1beta1}

Then, run the command:

 controller-gen crd:trivialVersions=true,preserveUnknownFields=false rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=deploy/crds/

👌 Testing your changes

After you have packaged a new version, test out your Operator on both OpenShift 4.8 and if available, an OpenShift 4.9 release candidate.

By using OperatorSDK (latest versions)

---

Testing bundles

To test and deploy the Operator with OLM, make sure the bundle image is present in a registry, operator-sdk run bundle can create a pod to serve that bundle to OLM via a Subscription, along with other OLM objects, ephemerally, e.g.:

$ operator-sdk run bundle <some-registry>/memcached-operator-bundle:v0.0.1

Note: If testing a bundle whose image will be hosted in a registry that is private and/or has a custom CA, these configuration steps must be complete.

Upgrading a bundle to a newer version

You can also use the operator-sdk run bundle-upgrade command with your newer version of the bundle image to upgrade an existing operator bundle deployed on the cluster. For example, to upgrade the previously deployed memcached-operator bundle from version 0.0.1 to 0.0.2, run the following:

$ operator-sdk run bundle-upgrade <some-registry>/memcached-operator-bundle:v0.0.2

---

Upgrading a bundle that was installed traditionally using OLM

If you want to test your Operator bundle upgrade even if it was originally deployed using OLM without using the run bundle command.

You can create a CatalogSource and deploy an Operator bundle traditionally using OLM and then upgrade the Operator bundle to a newer version. See more details.

Double-checking

To ensure you are not using a deprecated API anymore, you should see the alerts go away, and the oc command should show no usage:

$ oc get apirequestcount.apiserver.openshift.io/validatingwebhookconfigurations.v1beta1.admissionregistration.k8s.io
NAME                              REMOVEDINRELEASE   CURRENTHOUR   LAST24H
validatingwebhookconfigurations   1.22               4

[QuingKhaos]

Adding kubernetes-sigs/controller-runtime#1161 as a reference if an operator is utilizing it, it needs to be at least updated to controller-runtime@>=0.7.0 which includes the fix to accept the AdmissionReviewVersion v1