Couldn''t discover a KubeVirt Image for release image "quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750": no kubevirt vm disk image present in release' #1647

joaoacpjr · 2023-06-27T22:16:49Z

joaoacpjr
Jun 27, 2023

Hello!

I have an OKD 4.13 cluster installed on an Openstack, the Hosted Control Plane feature is enabled. I'm trying to create a hosted cluster using hypershift.

command used to create the hosted cluster:

hypershift create cluster kubevirt \
--name example2 \
--memory 6Gi \
--cores 2 \
--release-image=quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750 \
--root-volume-size 32 \
--node-pool-replicas=2 \
--root-volume-storage-class cinder-ssd-az-nova-csi \
--base-domain okd2.jacpjr.dev.br \
--external-dns-domain example2.okd2.jacpjr.dev.br \
--pull-secret pull-secret.txt

The hosted cluster status:

oc get hostedcluster example2 -n clusters                             
NAME       VERSION   KUBECONFIG                  PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
example2             example2-admin-kubeconfig   Partial    True        False         The hosted control plane is available

oc get hostedcluster example2 -n clusters -o yaml                     
apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  creationTimestamp: "2023-06-27T21:09:52Z"
  finalizers:
  - hypershift.openshift.io/finalizer
  generation: 3
  labels:
    hypershift.openshift.io/auto-created-for-infra: example2-v85dt
  name: example2
  namespace: clusters
  resourceVersion: "16882160"
  uid: 46535c62-f5be-4198-875e-d72fc25c6e0c
spec:
  autoscaling: {}
  clusterID: 8c59d46a-e097-456d-8616-009b8bb1c3e5
  controllerAvailabilityPolicy: SingleReplica
  dns:
    baseDomain: okd2.jacpjr.dev.br
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
        type: PersistentVolume
    managementType: Managed
  fips: false
  infraID: example2-v85dt
  infrastructureAvailabilityPolicy: SingleReplica
  issuerURL: https://kubernetes.default.svc
  networking:
    clusterNetwork:
    - cidr: 10.132.0.0/14
    networkType: OVNKubernetes
    serviceNetwork:
    - cidr: 172.31.0.0/16
  olmCatalogPlacement: management
  platform:
    kubevirt:
      generateID: zs4x6n4cgs
    type: KubeVirt
  pullSecret:
    name: example2-pull-secret
  release:
    image: quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750
  secretEncryption:
    aescbc:
      activeKey:
        name: example2-etcd-encryption-key
    type: aescbc
  services:
  - service: APIServer
    servicePublishingStrategy:
      type: LoadBalancer
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: Ignition
    servicePublishingStrategy:
      type: Route
  - service: OVNSbDb
    servicePublishingStrategy:
      type: Route
  sshKey: {}
status:
  conditions:
  - lastTransitionTime: "2023-06-27T21:10:48Z"
    message: Reconciliation completed succesfully
    observedGeneration: 3
    reason: ReconciliatonSucceeded
    status: "True"
    type: ReconciliationSucceeded
  - lastTransitionTime: "2023-06-27T21:14:45Z"
    message: 'Unable to apply 4.13.0-0.okd-2023-06-24-145750: an unknown error has
      occurred: MultipleErrors'
    observedGeneration: 3
    reason: MultipleErrors
    status: "True"
    type: ClusterVersionProgressing
  - lastTransitionTime: "2023-06-27T21:14:45Z"
    message: Payload loaded version="4.13.0-0.okd-2023-06-24-145750" image="quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750"
      architecture="amd64"
    observedGeneration: 3
    reason: PayloadLoaded
    status: "True"
    type: ClusterVersionReleaseAccepted
  - lastTransitionTime: "2023-06-27T21:09:54Z"
    message: Condition not found in the CVO.
    observedGeneration: 3
    reason: StatusUnknown
    status: Unknown
    type: ClusterVersionUpgradeable
  - lastTransitionTime: "2023-06-27T21:14:45Z"
    message: ""
    observedGeneration: 3
    reason: FromClusterVersion
    status: "False"
    type: ClusterVersionAvailable
  - lastTransitionTime: "2023-06-27T21:43:49Z"
    message: |-
      Multiple errors are preventing progress:
      * Cluster operators console, dns, image-registry, ingress, insights, kube-storage-version-migrator, monitoring, network, node-tuning, openshift-samples, service-ca are not available
      * Could not update machineconfig "99-okd-master-disable-mitigations" (418 of 583): the server does not recognize this resource, check extension API servers
      * Could not update machineconfig "99-okd-master-disable-mitigations" (451 of 583): the server does not recognize this resource, check extension API servers
    observedGeneration: 3
    reason: MultipleErrors
    status: "False"
    type: ClusterVersionSucceeding
  - lastTransitionTime: "2023-06-27T21:16:25Z"
    message: The hosted cluster is not degraded
    observedGeneration: 3
    reason: AsExpected
    status: "False"
    type: Degraded
  - lastTransitionTime: "2023-06-27T21:13:02Z"
    message: ""
    observedGeneration: 3
    reason: QuorumAvailable
    status: "True"
    type: EtcdAvailable
  - lastTransitionTime: "2023-06-27T21:13:37Z"
    message: Kube APIServer deployment is available
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: KubeAPIServerAvailable
  - lastTransitionTime: "2023-06-27T21:12:05Z"
    message: All is well
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: InfrastructureReady
  - lastTransitionTime: "2023-06-27T21:09:54Z"
    message: External DNS is not configured
    observedGeneration: 3
    reason: StatusUnknown
    status: Unknown
    type: ExternalDNSReachable
  - lastTransitionTime: "2023-06-27T21:10:57Z"
    message: Configuration passes validation
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: ValidHostedControlPlaneConfiguration
  - lastTransitionTime: "2023-06-27T21:09:54Z"
    message: Condition not found in the HCP
    observedGeneration: 3
    reason: StatusUnknown
    status: Unknown
    type: ValidReleaseInfo
  - lastTransitionTime: "2023-06-27T21:13:37Z"
    message: The hosted control plane is available
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: Available
  - lastTransitionTime: "2023-06-27T21:09:54Z"
    message: Configuration passes validation
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: ValidConfiguration
  - lastTransitionTime: "2023-06-27T21:09:54Z"
    message: HostedCluster is supported by operator configuration
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: SupportedHostedCluster
  - lastTransitionTime: "2023-06-27T21:14:46Z"
    message: Ignition server deployment is available
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: IgnitionEndpointAvailable
  - lastTransitionTime: "2023-06-27T21:09:54Z"
    message: Reconciliation active on resource
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: ReconciliationActive
  - lastTransitionTime: "2023-06-27T21:09:54Z"
    message: Release image is valid
    observedGeneration: 3
    reason: AsExpected
    status: "True"
    type: ValidReleaseImage
  - lastTransitionTime: "2023-06-27T21:09:54Z"
    message: HostedCluster is at expected version
    observedGeneration: 3
    reason: AsExpected
    status: "False"
    type: Progressing
  controlPlaneEndpoint:
    host: 81.0.223.231
    port: 6443
  ignitionEndpoint: ignition-server-clusters-example2.apps.okd2.jacpjr.dev.br
  kubeadminPassword:
    name: example2-kubeadmin-password
  kubeconfig:
    name: example2-admin-kubeconfig
  oauthCallbackURLTemplate: https://oauth-clusters-example2.apps.okd2.jacpjr.dev.br:443/oauth2callback/[identity-provider-name]
  version:
    availableUpdates: null
    desired:
      image: quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750
      version: 4.13.0-0.okd-2023-06-24-145750
    history:
    - completionTime: null
      image: quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750
      startedTime: "2023-06-27T21:14:19Z"
      state: Partial
      verified: false
      version: 4.13.0-0.okd-2023-06-24-145750
    observedGeneration: 1

example2 nodepool:

oc get nodepool example2 -n clusters -o yaml 
apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  creationTimestamp: "2023-06-27T21:09:53Z"
  finalizers:
  - hypershift.openshift.io/finalizer
  generation: 1
  labels:
    hypershift.openshift.io/auto-created-for-infra: example2-v85dt
  name: example2
  namespace: clusters
  ownerReferences:
  - apiVersion: hypershift.openshift.io/v1beta1
    kind: HostedCluster
    name: example2
    uid: 46535c62-f5be-4198-875e-d72fc25c6e0c
  resourceVersion: "16826875"
  uid: 42832d53-4565-4df0-8868-ea64b1dd332e
spec:
  arch: amd64
  clusterName: example2
  management:
    autoRepair: false
    replace:
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 0
      strategy: RollingUpdate
    upgradeType: Replace
  nodeDrainTimeout: 0s
  platform:
    kubevirt:
      compute:
        cores: 2
        memory: 6Gi
      rootVolume:
        persistent:
          size: 32Gi
          storageClass: cinder-ssd-az-nova-csi
        type: Persistent
    type: KubeVirt
  release:
    image: quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750
  replicas: 2
status:
  conditions:
  - lastTransitionTime: "2023-06-27T21:09:55Z"
    observedGeneration: 1
    reason: AsExpected
    status: "False"
    type: AutoscalingEnabled
  - lastTransitionTime: "2023-06-27T21:09:55Z"
    observedGeneration: 1
    reason: AsExpected
    status: "True"
    type: UpdateManagementEnabled
  - lastTransitionTime: "2023-06-27T21:09:55Z"
    message: 'Couldn''t discover a KubeVirt Image for release image "quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750":
      no kubevirt vm disk image present in release'
    observedGeneration: 1
    reason: ValidationFailed
    status: "False"
    type: ValidPlatformImage

oc get nodes -o wide 
NAME                        STATUS   ROLES                  AGE   VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION           CONTAINER-RUNTIME
okd2-l6npc-master-0         Ready    control-plane,master   8d    v1.26.5+7a891f0   10.0.1.99     <none>        Fedora CoreOS 38.20230514.3.0   6.2.15-300.fc38.x86_64   cri-o://1.26.0
okd2-l6npc-master-1         Ready    control-plane,master   8d    v1.26.5+7a891f0   10.0.2.189    <none>        Fedora CoreOS 38.20230514.3.0   6.2.15-300.fc38.x86_64   cri-o://1.26.0
okd2-l6npc-master-2         Ready    control-plane,master   8d    v1.26.5+7a891f0   10.0.3.99     <none>        Fedora CoreOS 38.20230514.3.0   6.2.15-300.fc38.x86_64   cri-o://1.26.0
okd2-l6npc-worker-0-bzl28   Ready    worker                 8d    v1.26.5+7a891f0   10.0.1.75     <none>        Fedora CoreOS 38.20230514.3.0   6.2.15-300.fc38.x86_64   cri-o://1.26.0
okd2-l6npc-worker-0-hpns7   Ready    worker                 8d    v1.26.5+7a891f0   10.0.1.209    <none>        Fedora CoreOS 38.20230514.3.0   6.2.15-300.fc38.x86_64   cri-o://1.26.0
okd2-l6npc-worker-0-zbcq4   Ready    worker                 8d    v1.26.5+7a891f0   10.0.3.70     <none>        Fedora CoreOS 38.20230514.3.0   6.2.15-300.fc38.x86_64   cri-o://1.26.0

Note: In another test I tried using OCP image quay.io/openshift-release-dev/ocp-release:4.12.21-multi and I got the same message about nodepool:

oc get nodepool example -n clusters -o yaml | grep message: -A 1
message: 'Couldn't discover a KubeVirt Image for the release image "quay.io/openshift-release-dev/ocp-release:4.13.4-multi" as there is no kubevirt vm disk image present in the release.'

vrutkovs · 2023-06-28T08:27:17Z

vrutkovs
Jun 28, 2023
Maintainer

OCP image issues should be reported to https://issues.redhat.com/browse/OCPBUGS

As for OKD bug it seems control plane has rolled out, but not the nodepool. IIUC hypershift uses machine-os-images image to find platform image, which is a copy from installer metadata - https://github.com/openshift/installer/blob/master/data/data/coreos/fcos.json. Unlike rhcos.json there is no kubevirt section, so we'll need:

a procedure to prepare this image from FCOS metadata and update FCOS pipeline to build the image
update https://github.com/coreos/fedora-coreos-streams/tree/main/streams so that we could bump installer metadata and have machine-os-images rebuilt

@andfasano do you know anything about KubeVirt Hypershift images?

2 replies

andfasano Jun 28, 2023
Collaborator

I'm afraid I won't be of a great help, as I'm not an expert of the KubeVirt Hypershift image. But from the bug description your analysis makes sense to me (due the lack of a kubevirt section in the embedded installer metadata) (and the same will be probably required for the future scos.json I guess)

joaoacpjr Jun 28, 2023
Author

Thank you for your response @vrutkovs! Is there a way where I can fix it locally on my cluster?

itmwiw · 2023-10-20T15:20:56Z

itmwiw
Oct 20, 2023

Hello,
I'm also trying to use Hypershift with OKD. I managed to work around the image issue by setting 'spec.platform.kubevirt.rootVolume.diskImage.containerDiskImage' to 'quay.io/fedora/fedora-coreos-kubevirt:stable' in the nodepool CR. Now, the VMs are created, and the nodes are marked as 'Ready.' Additionally, oc get co shows all operators as 'AVAILABLE':

NAME                                       VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console                                    4.13.0-0.okd-2023-09-30-084937   True        False         False      16s
csi-snapshot-controller                    4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
dns                                        4.13.0-0.okd-2023-09-30-084937   True        False         False      2m12s
image-registry                             4.13.0-0.okd-2023-09-30-084937   True        False         False      2m25s
ingress                                    4.13.0-0.okd-2023-09-30-084937   True        False         False      2m7s
insights                                   4.13.0-0.okd-2023-09-30-084937   True        False         False      3m20s
kube-apiserver                             4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
kube-controller-manager                    4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
kube-scheduler                             4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
kube-storage-version-migrator              4.13.0-0.okd-2023-09-30-084937   True        False         False      2m43s
monitoring                                 4.13.0-0.okd-2023-09-30-084937   True        False         False      66s
network                                    4.13.0-0.okd-2023-09-30-084937   True        False         False      3m39s
node-tuning                                4.13.0-0.okd-2023-09-30-084937   True        False         False      6m48s
openshift-apiserver                        4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
openshift-controller-manager               4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
openshift-samples                          4.13.0-0.okd-2023-09-30-084937   True        False         False      111s
operator-lifecycle-manager                 4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
operator-lifecycle-manager-catalog         4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
operator-lifecycle-manager-packageserver   4.13.0-0.okd-2023-09-30-084937   True        False         False      50m
service-ca                                 4.13.0-0.okd-2023-09-30-084937   True        False         False      3m18s
storage                                    4.13.0-0.okd-2023-09-30-084937   True        False         False      50m

However, I still encounter an error related to the '99-okd-master-disable-mitigations' machineconfig. The exact error is as follow:

E1020 15:06:53.905739       1 sync_worker.go:652] unable to synchronize image (waiting 2m36.124787484s): Multiple errors are preventing progress:
* Could not update machineconfig "99-okd-master-disable-mitigations" (418 of 584): the server does not recognize this resource, check extension API servers
* Could not update machineconfig "99-okd-master-disable-mitigations" (451 of 584): the server does not recognize this resource, check extension API servers

I suppose, at some point, the installation program attempts to apply the machineconfig to the cluster, even though the cluster does not have the corresponding CRD. In fact, machineconfigs are managed from the 'hub' cluster where the control plane is located.
@vrutkovs, can you think of a workaround for this behavior? Why does the installation program attempt to apply this machineconfig at this point? If it is a control check that triggers it, perhaps I can add a machine config as described here to prevent the 'apply' from being triggered?
Regards,
Tarik

4 replies

vrutkovs Oct 20, 2023
Maintainer

the installation program attempts to apply the machineconfig to the cluster, even though the cluster does not have the corresponding CRD

~~So the hub cluster doesn't have MachineConfig CRD? Is it plain k8s?~~
Nevermind, it seems OKD specific - I can reproduce that with Hypershift on AWS too

Why does the installation program attempt to apply this machineconfig at this point?

By default FCOS boots with Spectre mitigations on. We want to disable it to avoid bumping CPU requirements.

perhaps I can add a machine config as described here to prevent the 'apply' from being triggered?

That might work, but it feels like a weird workaround

itmwiw Oct 21, 2023

The 'hub cluster' does have a MachineConfig, as it is a typical OKD cluster. When creating 'hosted clusters' using Hypershift, the control plane is deployed as pods, and the worker nodes are deployed as VMs, which could be kubevirt, AWS, or other cloud providers. Because the control plane resides as pods in the 'hub cluster', certain components and CRDs are disabled for the 'hosted cluster'. I suppose that is the reason why the MachineConfigs do not exist in the 'hosted cluster,' although I am not entirely certain.

vrutkovs Oct 22, 2023
Maintainer

Right, that seems OKD FCOS specific, it doesn't happen on OKD SCOS.

Could you file a separate OKD issue for that?

itmwiw Oct 22, 2023

Ok I'll do it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Couldn''t discover a KubeVirt Image for release image "quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750": no kubevirt vm disk image present in release' #1647

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Couldn''t discover a KubeVirt Image for release image "quay.io/openshift/okd:4.13.0-0.okd-2023-06-24-145750": no kubevirt vm disk image present in release' #1647

joaoacpjr Jun 27, 2023

Replies: 2 comments · 6 replies

vrutkovs Jun 28, 2023 Maintainer

andfasano Jun 28, 2023 Collaborator

joaoacpjr Jun 28, 2023 Author

itmwiw Oct 20, 2023

vrutkovs Oct 20, 2023 Maintainer

itmwiw Oct 21, 2023

vrutkovs Oct 22, 2023 Maintainer

itmwiw Oct 22, 2023

joaoacpjr
Jun 27, 2023

Replies: 2 comments 6 replies

vrutkovs
Jun 28, 2023
Maintainer

andfasano Jun 28, 2023
Collaborator

joaoacpjr Jun 28, 2023
Author

itmwiw
Oct 20, 2023

vrutkovs Oct 20, 2023
Maintainer

vrutkovs Oct 22, 2023
Maintainer