[OKD-SCOS v4.16] Upgrade from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-07-11-155249 fails: invalid config #1971

titou10titou10 · 2024-07-12T14:58:58Z

titou10titou10
Jul 12, 2024

(this is in relation to discussion #1960)

Context

Trying to upgrade a SNO (Single Node) v4.15.0-0.okd-scos-2024-01-18-223523 cluster to v4.16.0-0.okd-scos-2024-07-11-155249

installed cluster:
- SNO OKD-SCOS v4.15.0-0.okd-scos-2024-01-18-223523 (an "Accepted" version)
- on baremetal with UPI
- using agent based installer (ABI)
- on vm: 40G RAM, 100G SSD, 10 vCPUS on Proxmox (qemu)
upgrading to 4.16.0-0.okd-scos-2024-07-11-155249 (not "accepted" as there is no such versions yet)

I also have tried for a multi-node cluster, it fails the same way

I've encountered the exact same problem in many other scenarios and many different versions (upgrade from OKD-FCOS v4.15 to OKD-SCOS v4.16, from OKD-FCOS v4.15 to OKD-SCOS v4.17..)

Summary

It fails with the following error:

Could not update customresourcedefinition "infrastructures.config.openshift.io"
  (47 of 895): the object is invalid, possibly due to local cluster configuration

And in cluster-version-operator logs:

spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[ingressIPs].x-kubernetes-validations[0].rule: 
Invalid value: apiextensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", M
Message:"ingressIPs must contain at most one IPv4 address and at most one IPv6 address", MessageExpression:"", 
Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:24: undeclared reference to 'isIP' (in container '')

So it tries to validate somevsphereparameter while this is a baremetal UPI install ????

Details

Base cluster information:

oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        False         37s     Cluster version is 4.15.0-0.okd-scos-2024-01-18-223523

oc get nodes -o wide
NAME      STATUS   ROLES                         AGE   VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
okd-sno   Ready    control-plane,master,worker   32m   v1.28.3+20a5764   192.168.5.71   <none>        CentOS Stream CoreOS 415.9.202401181453-0   5.14.0-408.el9.x86_64   cri-o://1.28.2

oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      11m
baremetal                                  4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
cloud-controller-manager                   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
cloud-credential                           4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      21m
cluster-autoscaler                         4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
config-operator                            4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
console                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      86s
control-plane-machine-set                  4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
csi-snapshot-controller                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
dns                                        4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      26m
etcd                                       4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
image-registry                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      14m
ingress                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
insights                                   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
kube-apiserver                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      18m
kube-controller-manager                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      19m
kube-scheduler                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
kube-storage-version-migrator              4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
machine-api                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
machine-approver                           4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      22m
machine-config                             4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
marketplace                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
monitoring                                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      15m
network                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      29m
node-tuning                                4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
openshift-apiserver                        4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      15m
openshift-controller-manager               4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      14m
openshift-samples                          4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      18m
operator-lifecycle-manager                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
operator-lifecycle-manager-catalog         4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      27m
operator-lifecycle-manager-packageserver   4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      19m
service-ca                                 4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m
storage                                    4.15.0-0.okd-scos-2024-01-18-223523   True        False         False      28m

Force upgrade to v4.16.0-0.okd-scos-2024-07-11-155249

oc adm upgrade --force --allow-explicit-upgrade --to-image=registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249

warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249

After a few minutes, the upgrade fails:

oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        True          2m7s    Unable to apply 4.16.0-0.okd-scos-2024-07-11-155249: some cluster configuration is invalid


oc get clusterversion -o yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2024-07-12T14:00:23Z"
    generation: 3
    name: version
    resourceVersion: "27690"
    uid: 5710e759-147c-48bf-abfc-fe4579f56ee7
  spec:
    channel: stable-scos-4
    clusterID: 1e066a1d-66a1-4115-b7d4-aa80d71af058
    desiredUpdate:
      architecture: ""
      force: true
      image: registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249
      version: ""
    upstream: https://amd64.origin.releases.ci.openshift.org/graph
  status:
    availableUpdates: null
    capabilities:
      enabledCapabilities:
      - Build
      - CSISnapshot
      - CloudControllerManager
      - CloudCredential
      - Console
      - DeploymentConfig
      - ImageRegistry
      - Ingress
      - Insights
      - MachineAPI
      - NodeTuning
      - OperatorLifecycleManager
      - Storage
      - baremetal
      - marketplace
      - openshift-samples
      knownCapabilities:
      - Build
      - CSISnapshot
      - CloudControllerManager
      - CloudCredential
      - Console
      - DeploymentConfig
      - ImageRegistry
      - Ingress
      - Insights
      - MachineAPI
      - NodeTuning
      - OperatorLifecycleManager
      - Storage
      - baremetal
      - marketplace
      - openshift-samples
    conditions:
    - lastTransitionTime: "2024-07-12T14:00:31Z"
      status: "True"
      type: RetrievedUpdates
    - lastTransitionTime: "2024-07-12T14:45:52Z"
      message: Capabilities match configured spec
      reason: AsExpected
      status: "False"
      type: ImplicitlyEnabledCapabilities
    - lastTransitionTime: "2024-07-12T14:00:31Z"
      message: Payload loaded version="4.16.0-0.okd-scos-2024-07-11-155249" image="registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249"
        architecture="amd64"
      reason: PayloadLoaded
      status: "True"
      type: ReleaseAccepted
    - lastTransitionTime: "2024-07-12T14:40:30Z"
      message: Done applying 4.15.0-0.okd-scos-2024-01-18-223523
      status: "True"
      type: Available
    - lastTransitionTime: "2024-07-12T14:47:22Z"
      message: 'Could not update customresourcedefinition "infrastructures.config.openshift.io"
        (47 of 895): the object is invalid, possibly due to local cluster configuration'
      reason: UpdatePayloadResourceInvalid
      status: "True"
      type: Failing
    - lastTransitionTime: "2024-07-12T14:45:38Z"
      message: 'Unable to apply 4.16.0-0.okd-scos-2024-07-11-155249: some cluster
        configuration is invalid'
      reason: UpdatePayloadResourceInvalid
      status: "True"
      type: Progressing
    desired:
      image: registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249
      version: 4.16.0-0.okd-scos-2024-07-11-155249
    history:
    - acceptedRisks: |-
        Target release version="" image="registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249" cannot be verified, but continuing anyway because the update was forced: release images that are not accessed via digest cannot be verified
        Precondition "ClusterVersionRecommendedUpdate" failed because of "UnknownUpdate": RetrievedUpdates=True (), so the update from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-07-11-155249 is probably neither recommended nor supported.
      completionTime: null
      image: registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-11-155249
      startedTime: "2024-07-12T14:45:38Z"
      state: Partial
      verified: false
      version: 4.16.0-0.okd-scos-2024-07-11-155249
    - completionTime: "2024-07-12T14:40:30Z"
      image: quay.io/okd/scos-release@sha256:ca26e01206c31fa9fb74f611cb5258aea48c03b7ce146251d86b984c1ee72942
      startedTime: "2024-07-12T14:00:31Z"
      state: Completed
      verified: false
      version: 4.15.0-0.okd-scos-2024-01-18-223523
    observedGeneration: 3
    versionHash: iZdIgxduII8=
 kind: List
 metadata:
   resourceVersion: ""

infrastructures.config.openshift.io:

oc get infrastructures.config.openshift.io cluster -o yaml

apiVersion: config.openshift.io/v1
kind: Infrastructure
metadata:
  creationTimestamp: "2024-07-12T14:00:17Z"
  generation: 1
  name: cluster
  resourceVersion: "531"
  uid: baae0b43-feff-4c31-a94d-120c595435cb
spec:
  cloudConfig:
    name: ""
  platformSpec:
    type: None
status:
  apiServerInternalURI: https://api-int.okd-sno.denis.prive:6443
  apiServerURL: https://api.okd-sno.denis.prive:6443
  controlPlaneTopology: SingleReplica
  cpuPartitioning: None
  etcdDiscoveryDomain: ""
  infrastructureName: okd-sno-pbbjs
  infrastructureTopology: SingleReplica
  platform: None
  platformStatus:
    type: None

[UPDATED]
After some investigation, theopenshift-cluster-version/cluster-version-operatorpod logs reports related errors

oc logs cluster-version-operator-674f757d7d-gj2hs -n openshift-cluster-version
{...}
  I0712 14:45:54.939774       1 sync_worker.go:1014] Done syncing for customresourcedefinition "images.config.openshift.io" (45 of 895)
  I0712 14:45:54.939956       1 sync_worker.go:999] Running sync for customresourcedefinition "imagetagmirrorsets.config.openshift.io" (46 of 895)
  I0712 14:45:54.998132       1 sync_worker.go:1014] Done syncing for customresourcedefinition "imagetagmirrorsets.config.openshift.io" (46 of 895)
  I0712 14:45:54.998268       1 sync_worker.go:999] Running sync for customresourcedefinition "infrastructures.config.openshift.io" (47 of 895)
  E0712 14:45:55.517832       1 task.go:122] error running apply for customresourcedefinition "infrastructures.config.openshift.io" (47 of 895): CustomResourceDefinition.apiextension
  s.k8s.io "infrastructures.config.openshift.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[apiServerIntern
  alIPs].items.x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"isIP(self)", Message:"value must be a valid IP address", MessageExpression:"", Reas
  on:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:5: undeclared reference to 'isIP' (in container '')
   | isIP(self)
   | ....^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[apiServerInternalIPs].x-kubernetes-validations[0].rule: Invalid v
  alue: apiextensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", Message:"apiServerInternalIPs mus
  t contain at most one IPv4 address and at most one IPv6 address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR
  : <input>:1:24: undeclared reference to 'isIP' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | .......................^
  ERROR: <input>:1:41: undeclared reference to 'isIP' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ........................................^
  ERROR: <input>:1:55: undeclared reference to 'ip' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ......................................................^
  ERROR: <input>:1:71: undeclared reference to 'family' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ......................................................................^
  ERROR: <input>:1:79: undeclared reference to 'ip' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ..............................................................................^
  ERROR: <input>:1:95: undeclared reference to 'family' (in container '')
   | size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
   | ..............................................................................................^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].proper
  ties[vsphere].properties[ingressIPs].items.x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"isIP(self)", Message:"value must be a valid IP addres
  s", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:5: undeclared reference to 'isIP' (in container ''
  )
   | isIP(self)
   | ....^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[ingressIPs].x-kubernetes-validations[0].rule: Invalid value: apie
  xtensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", Message:"ingressIPs must contain at most on
  e IPv4 address and at most one IPv6 address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:24: unde
  clared reference to 'isIP' (in container '')

So it seems it is trying to validate somevsphereparameter while this is a baremetal UPI install ????

spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[vsphere].properties[ingressIPs].x-kubernetes-validations[0].rule: 
Invalid value: apiextensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", M
Message:"ingressIPs must contain at most one IPv4 address and at most one IPv6 address", MessageExpression:"", 
Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:24: undeclared reference to 'isIP' (in container '')

madpearl · 2024-07-22T10:04:46Z

madpearl
Jul 22, 2024

There is acceptable version of 4.16.0-0.okd-scos-2024-07-21-033952 , you can check it, but I thing that you will get the same error, because the difference between 4.15.0-0.okd-scos-2024-01-18-223523 are too many!

0 replies

aleskandro · 2024-07-22T22:01:13Z

aleskandro
Jul 22, 2024
Maintainer

Hi all,
this error happens because some new validation filters have been set in the CRDs that the CVO applies before updating to the kube-apiserver version supporting them.

Can you help test this procedure on your platforms?

I tested the same path of this thread's title (4.15.0-0.okd-scos-2024-01-18-223523 -> 4.16.0-0.okd-scos-2024-07-11-155249) in a fresh 4.15 OKD on AWS, and it went fine.

oc adm upgrade --to-image registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952 --force --allow-explicit-upgrades
Wait for the error to come up:

ERROR: <input>:1:5: undeclared reference to 'isIP'

oc scale --replicas=0 deployments/cluster-version-operator -n openshift-cluster-version. The upgrade will stop here.
oc adm release info --image-for=hyperkube registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952
oc adm release info --image-for=cluster-kube-apiserver-operator registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952
Force the kube-apiserver upgrade by patching the kube-apiserver-operator. oc edit -n openshift-kube-apiserver-operator deployments/kube-apiserver-operator and change:

the image of the kube-apiserver-operator container to the one given by point (6)
the value of the OPERATOR_IMAGE env var of the kube-apiserver-operator container to the one given by point (6)
the value of the IMAGE env var of the kube-apiserver-operator container to the one given by point (5) [hyperkube]
the value of the OPERAND_IMAGE_VERSION env var to 1.29.6
keep the OPERATOR_IMAGE_VERSION as is (4.15.xxx...)

Wait for the new kube-apiserver-operator and kube-apiserver to be deployed (all the replicas if non-SNO) by watching at oc get pods -n openshift-kube-apiserver-operator && oc get pods -n openshift-kube-apiserver
Once all the replicas of the new kube-apiserver are ready, scale out the CVO: oc scale --replicas=1 deployments/cluster-version-operator -n openshift-cluster-version
the upgrade should complete. You can follow with:

watch oc get co && oc get nodes && oc get mcp && oc get clusterversion

8 replies

Elyytscha Sep 26, 2024

i can confirm this is working for me too, i upgraded successfully from okd 4.15 to 4.16 with this tutorial

JOECHONG0420 Sep 30, 2024

When I upgrade from 4.15.0-0.okd-2024-03-10-010116 to 4.16.0-0.okd-scos-2024-08-21-155613
My cluster operator "authentication" and "openshift-apiserver" show APIServicesAvailable: PreconditionNotReady

alexminder Sep 30, 2024

When I upgrade from 4.15.0-0.okd-2024-03-10-010116 to 4.16.0-0.okd-scos-2024-08-21-155613 My cluster operator "authentication" and "openshift-apiserver" show APIServicesAvailable: PreconditionNotReady

Same for me with 2024-08-21 and 2024-09-24

JOECHONG0420 Oct 2, 2024

it seem the health check in openshift-apiserver failed.

JOECHONG0420 Oct 3, 2024

I also try the version 4.16.0-0.okd-scos-2024-09-27-110344, it appear the same error.
In openshift-apiserver namespace, the health check of apiserver failed with error message below. How can I fix this error?
[-]poststarthook/authorization.openshift.io-bootstrapclusterroles failed: reason withheld [-]poststarthook/authorization.openshift.io-ensurenodebootstrap-sa failed: reason withheld

titou10titou10 · 2024-07-22T23:43:57Z

titou10titou10
Jul 22, 2024
Author

oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        False         7h45m   Cluster version is 4.15.0-0.okd-scos-2024-01-18-223523

1. 
oc adm upgrade --to-image registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952 --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952

2.
oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.0-0.okd-scos-2024-01-18-223523   True        True          3m27s   Unable to apply 4.16.0-0.okd-scos-2024-07-21-033952: some cluster configuration is invalid

3,
oc logs -n openshift-cluster-version cluster-version-operator-5987fd6b6-cfplw
E0722 23:34:06.714276       1 task.go:122] error running apply for customresourcedefinition "infrastructures.config.openshift.io" (47 of 895): CustomResourceDefinition.apiextensions.k8s.io "infrastructures.config.openshift.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[openstack].properties[machineNetworks].items.x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"isCIDR(self)", Message:"value must be a valid CIDR network address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:7: undeclared reference to 'isCIDR' (in container '')
| isCIDR(self)
| ......^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[openstack].properties[apiServerInternalIPs].items.x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"isIP(self)", Message:"value must be a valid IP address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:5: undeclared reference to 'isIP' (in container '')
| isIP(self)
| ....^, spec.validation.openAPIV3Schema.properties[spec].properties[platformSpec].properties[openstack].properties[apiServerInternalIPs].x-kubernetes-validations[0].rule: Invalid value: apiextensions.ValidationRule{Rule:"size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true", Message:"apiServerInternalIPs must contain at most one IPv4 address and at most one IPv6 address", MessageExpression:"", Reason:(*apiextensions.FieldValueErrorReason)(nil), FieldPath:""}: compilation failed: ERROR: <input>:1:24: undeclared reference to 'isIP' (in container '')
| size(self) == 2 && isIP(self[0]) && isIP(self[1]) ? ip(self[0]).family() != ip(self[1]).family() : true
| .......................^

4. 
oc scale --replicas=0 deployments/cluster-version-operator -n openshift-cluster-version
deployment.apps/cluster-version-operator scaled
oc get pods -n openshift-cluster-version
NAME                   READY   STATUS      RESTARTS   AGE
version--pk2hp-l6tc6   0/1     Completed   0          5m23s

5. 
oc adm release info --image-for=hyperkube registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952
registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:2800d1392d9ef7759df244a709337d288ead95e75f439749f9fe57c347afe8b1

6. 
oc adm release info --image-for=cluster-kube-apiserver-operator registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-21-033952
registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:50194105e197b103e1af59efda006ed680f2e631c841d63abda7bf5444dcfca3

7. 
oc edit -n openshift-kube-apiserver-operator deployments/kube-apiserver-operator

  env:
    - name: IMAGE   
      value: registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:2800d1392d9ef7759df244a709337d288ead95e75f439749f9fe57c347afe8b1
    - name: OPERATOR_IMAGE
      value: registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:50194105e197b103e1af59efda006ed680f2e631c841d63abda7bf5444dcfca3
    - name: OPERAND_IMAGE_VERSION
      value: 1.29.6
    - name: OPERATOR_IMAGE_VERSION
      value: 4.15.0-0.okd-scos-2024-01-18-223523
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: registry.ci.openshift.org/origin/4.16-okd-scos-2024-07-21-033952@sha256:50194105e197b103e1af59efda006ed680f2e631c841d63abda7bf5444dcfca3

8.
oc get pods -n openshift-kube-apiserver-operator && oc get pods -n openshift-kube-apiserver | grep -v Completed
NAME                                       READY   STATUS    RESTARTS   AGE
kube-apiserver-operator-5dd99f6bd6-cv4bn   1/1     Running   0          3m31s
NAME                          READY   STATUS      RESTARTS   AGE
kube-apiserver-okd-sno        5/5     Running     0          78s

9.
oc scale --replicas=1 deployments/cluster-version-operator -n openshift-cluster-version

10.
oc get clusterversion
NAME      VERSION                               AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.0-0.okd-scos-2024-07-21-033952   True        False         8h      Cluster version is 4.16.0-0.okd-scos-2024-07-21-033952

oc get nodes -o wide
NAME      STATUS   ROLES                         AGE   VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                    KERNEL-VERSION          CONTAINER-RUNTIME
okd-sno   Ready    control-plane,master,worker   20h   v1.29.6+aba1e8d   192.168.5.71   <none>        CentOS Stream CoreOS 417.9.202407210037-0   5.14.0-480.el9.x86_64   cri-o://1.29.5

oc get co
NAME                                       VERSION                               AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      111m
baremetal                                  4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
cloud-controller-manager                   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
cloud-credential                           4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
cluster-autoscaler                         4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
config-operator                            4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
console                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
control-plane-machine-set                  4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
csi-snapshot-controller                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
dns                                        4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      8h
etcd                                       4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
image-registry                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
ingress                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
insights                                   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
kube-apiserver                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
kube-controller-manager                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
kube-scheduler                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
kube-storage-version-migrator              4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
machine-api                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
machine-approver                           4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
machine-config                             4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
marketplace                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
monitoring                                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
network                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
node-tuning                                4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
openshift-apiserver                        4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      151m
openshift-controller-manager               4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      115m
openshift-samples                          4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      9h
operator-lifecycle-manager                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
operator-lifecycle-manager-catalog         4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
operator-lifecycle-manager-packageserver   4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
service-ca                                 4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h
storage                                    4.16.0-0.okd-scos-2024-07-21-033952   True        False         False      20h

0 replies

titou10titou10 · 2024-07-23T12:19:27Z

titou10titou10
Jul 23, 2024
Author

So it's a success
Thanks

Will this fix be included in the next "Accepted" or "Stable" version for SCOS 4.16/4.17? Or the next stable v4.15 SCOS version?

0 replies

michael-willingham · 2024-07-27T05:27:53Z

michael-willingham
Jul 27, 2024

Has anyone gotten this guide to work on a single node cluster? After editing the kube-apiserver-operator, the actual kube-apiservers don't seem to update/pull in a new version. Re-scaling the cv operator leads to the same failure as before 😢

3 replies

madpearl Jul 27, 2024

You have to put the actual OKD build release in the "okd adm release info...", because old releases are deleted.

michael-willingham Jul 28, 2024

Yeah, I've tried with the latest accepted / actually available releases, same issue sadly.

michael-willingham Jul 28, 2024

For more clarity - I'm following this guide (for a SNO cluster - 4.15 fcos -> 4.15 scos -> 4.16 scos), and my kube-apiserver pods are never re-deploying with the updated image after modifying the kube-apiserver-operator. So I get the same CRD validation error in the cluster-version-operator. I started with a fresh 4.15 fcos install, and have done nothing previously besides the jump to 4.15 scos.

titou10titou10 · 2024-07-27T13:05:02Z

titou10titou10
Jul 27, 2024
Author

@michael-willingham if you re-read my initial post and the various comments, you'll see that I've done it on a single node cluster (aka SNO)

1 reply

michael-willingham Jul 28, 2024

Thanks! Totally missed that this was a SNO cluster.

titou10titou10 · 2024-07-31T16:49:48Z

titou10titou10
Jul 31, 2024
Author

Update from "stable" v4.15.0-0.okd-scos-2024-01-18-223523 to "next" v4.16.0-0.okd-scos-2024-07-29-184808 gives the same error ("'Could not update customresourcedefinition "infrastructures.config.openshift.io")
So the problem is still present in the latest SCOS versions

0 replies

madpearl · 2024-08-16T09:12:47Z

madpearl
Aug 16, 2024

I've upgraded to "next" 4.16.0-0.okd-scos-2024-08-01-132038 from stable 4.15.0-0.okd-scos-2024-01-18-223523 with the same procedure. I think it cannot be applied the openshift-kube-apiserver-operator config upgrade without going through all the steps of manual reconfiguration described from @aleskandro .

0 replies

globaldyne · 2024-09-30T13:00:24Z

globaldyne
Sep 30, 2024

Followed the guide above, tried to upgrade from 4.15.0-0.okd-2024-03-10-010116 to 4.16.0-0.okd-scos-2024-09-27-110344
Machine config daemon crash looping with
I0930 12:57:49.876855 9599 start.go:68] Version: machine-config-daemon-4.6.0-202006240615.p0-2842-g6a425abb-dirty (6a425abbe5e330f0e7bc0849f4927d4c55d0c9e5) E0930 12:57:49.876997 9599 rpm-ostree.go:276] Merged secret file does not exist; defaulting to cluster pull secret I0930 12:57:49.877022 9599 rpm-ostree.go:263] Linking ostree authfile to /var/lib/kubelet/config.json F0930 12:57:49.942414 9599 start.go:106] Failed to initialize single run daemon: error reading osImageURL from rpm-ostree: exit status 1

So the node upgrade not progressing.
Any ideas?

5 replies

GingerGeek Oct 1, 2024
Maintainer

Can you confirm what shape your OKD cluster is? Are you IPI/UPI? Single node/multi-node? Bare metal?

globaldyne Oct 1, 2024

6 nodes, Bare metal

Prashanth684 Oct 1, 2024
Maintainer

could you log into the problematic node and get the logs from there? which systemd units are failing?

globaldyne Oct 2, 2024

Can't provide any logs as i had to rebuild the cluster back to 4.15 FCOS
Although i've noticed the fsck systemd service was failing. Same issue occurs if i provision a bootstrap node directly in the latest 4.16 SCOS

tdemaret Oct 9, 2024

I have the same issue, maybe my logs can help. It is IPI deployment on Openstack :
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
okd-8kb4s-master-0 Ready master 2y135d v1.29.8+f10c92d 172.32.1.111 10.10.11.187 CentOS Stream CoreOS 418.9.202409241032-0 5.14.0-511.el9.x86_64 cri-o://1.29.5
okd-8kb4s-master-1 Ready master 2y135d v1.29.8+f10c92d 172.32.2.18 CentOS Stream CoreOS 418.9.202409241032-0 5.14.0-511.el9.x86_64 cri-o://1.29.5
okd-8kb4s-master-2 Ready master 2y135d v1.29.8+f10c92d 172.32.2.126 CentOS Stream CoreOS 418.9.202409241032-0 5.14.0-511.el9.x86_64 cri-o://1.29.5
okd-8kb4s-worker-415-0-nz5np Ready worker 4d20h v1.28.7+6e2789b 172.32.1.165 Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
okd-8kb4s-worker-415-0-svbj7 Ready,SchedulingDisabled worker 4d20h v1.29.8+f10c92d 172.32.1.173 CentOS Stream CoreOS 418.9.202409241032-0 5.14.0-511.el9.x86_64 cri-o://1.29.5
okd-8kb4s-worker-415-0-wgj46 Ready worker 4d17h v1.28.7+6e2789b 172.32.0.23 Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2
okd-8kb4s-worker-415-0-wlpxz Ready worker 4d17h v1.28.7+6e2789b 172.32.2.149 Fedora CoreOS 39.20240210.3.0 6.7.4-200.fc39.x86_64 cri-o://1.28.2

Any issue for masters, but it is not the same for worker node.
Uploading node-worker.log…

soubinan · 2024-11-20T10:40:39Z

soubinan
Nov 20, 2024

The fixes proposed here were not working for me
Actually, I was not so surprised to see some issues on the upgrade since it is a double change to deal with (lot of code changes + base OS change)
The easiest move (what I did) was to go from 4.15 okd to 4.15 okd-scos and then moving from the "new" 4.15 to 4.16
Sure it is double the work, but it is more reliable and without any workaround

2 replies

Elyytscha Nov 20, 2024

that are the exact version which worked for me, maybe this helps, i had to do #1971 (comment), at the switch to scos.

soubinan Nov 21, 2024

For me, the apiserver never got updated after my changes on the operator
So the upgrade was looping on the error until the upgrade fail

[OKD-SCOS v4.16] Upgrade from 4.15.0-0.okd-scos-2024-01-18-223523 to 4.16.0-0.okd-scos-2024-07-11-155249 fails: invalid config #1971

Context

Summary

Details

Replies: 10 comments · 19 replies

aleskandro Jul 22, 2024 Maintainer

titou10titou10 Jul 22, 2024 Author

titou10titou10 Jul 23, 2024 Author

titou10titou10 Jul 27, 2024 Author

titou10titou10 Jul 31, 2024 Author

GingerGeek Oct 1, 2024 Maintainer

Prashanth684 Oct 1, 2024 Maintainer

Replies: 10 comments 19 replies

aleskandro
Jul 22, 2024
Maintainer

titou10titou10
Jul 22, 2024
Author

titou10titou10
Jul 23, 2024
Author

titou10titou10
Jul 27, 2024
Author

titou10titou10
Jul 31, 2024
Author

GingerGeek Oct 1, 2024
Maintainer

Prashanth684 Oct 1, 2024
Maintainer