[Libvirt] Mirroring of images with imageContentSourcePolicy not working ?

[ObieBent]

Hello,

I try to mirror an image "community-operator-index:v4.12" from "registry.access.redhat.com" to my private internal registry.
Here is below the imageContentSourcePolicy file used to perform this task.

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"operator.openshift.io/v1alpha1","kind":"ImageContentSourcePolicy","metadata":{"annotations":{},"labels":{"operators.openshift.org/catalog":"true"},"name":"operator-0"},"spec":{"repositoryDigestMirrors":[{"mirrors":["registry.cluster-okd.buzz.lab:5000/operators/openshift-logging"],"source":"registry.redhat.io/openshift-logging"},{"mirrors":["registry.cluster-okd.buzz.lab:5000/operators/redhat"],"source":"registry.redhat.io/redhat"},{"mirrors":["registry.cluster-okd.buzz.lab:5000/operators/openshift-gitops-1"],"source":"registry.redhat.io/openshift-gitops-1"}]}}
  creationTimestamp: "2023-08-03T09:05:21Z"
  generation: 1
  labels:
    operators.openshift.org/catalog: "true"
  name: operator-0
  resourceVersion: "4180509"
  uid: 0cfc325c-bb28-43f2-9a7d-6507f7246b9e
spec:
  repositoryDigestMirrors:
  - mirrors:
    - registry.cluster-okd.buzz.lab:5000/operators/openshift-logging
    source: registry.redhat.io/openshift-logging
  - mirrors:
    - registry.cluster-okd.buzz.lab:5000/operators/redhat
    source: registry.redhat.io/redhat
  - mirrors:
    - registry.cluster-okd.buzz.lab:5000/operators/openshift-gitops-1
    source: registry.redhat.io/openshift-gitops-1

Based on this policy, the /etc/containers/registries.conf file of the cluster nodes is populated accordingly.

# cat /etc/containers/registries.conf
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
short-name-mode = ""

[[registry]]
  prefix = ""
  location = "gcr.io/kubebuilder"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/operators/kubebuilder"

[[registry]]
  prefix = ""
  location = "quay.io/opdev"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/operators/opdev"

[[registry]]
  prefix = ""
  location = "quay.io/openshift-community-operators"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/operators/openshift-community-operators"

[[registry]]
  prefix = ""
  location = "quay.io/openshift/okd"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/ocp4/okd"

[[registry]]
  prefix = ""
  location = "quay.io/openshift/okd-content"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/ocp4/okd"

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/redhat"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/operators/redhat"

[[registry]]
  prefix = ""
  location = "registry.redhat.io/openshift-gitops-1"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/operators/openshift-gitops-1"

[[registry]]
  prefix = ""
  location = "registry.redhat.io/openshift-logging"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/operators/openshift-logging"

[[registry]]
  prefix = ""
  location = "registry.redhat.io/redhat"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "registry.cluster-okd.buzz.lab:5000/operators/redhat"

The strange thing is when I try to pull the image with podman on one of my nodes, it does not even try to pull the mirrored image from the mirror I defined in the ImageContentSourcePolicy.

# podman pull --log-level debug registry.access.redhat.com/redhat/community-operator-index:v4.12
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level debug registry.access.redhat.com/redhat/community-operator-index:v4.12)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/containers/storage" with "/var/run/containers/storage" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] NewControl(/var/lib/containers/storage/overlay): nextProjectID = 3441329851
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=xfs, projectQuotaSupported=true, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 19
DEBU[0000] Pulling image registry.access.redhat.com/redhat/community-operator-index:v4.12 (policy: always)
DEBU[0000] Looking up image "registry.access.redhat.com/redhat/community-operator-index:v4.12" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "registry.access.redhat.com/redhat/community-operator-index:v4.12" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@a7185479d1426898e8d417574362b30bdaa7a05d8db114432ec13ee724388f03"
DEBU[0000] Found image "registry.access.redhat.com/redhat/community-operator-index:v4.12" as "registry.access.redhat.com/redhat/community-operator-index:v4.12" in local containers storage
DEBU[0000] Found image "registry.access.redhat.com/redhat/community-operator-index:v4.12" as "registry.access.redhat.com/redhat/community-operator-index:v4.12" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage]@a7185479d1426898e8d417574362b30bdaa7a05d8db114432ec13ee724388f03)
DEBU[0000] exporting opaque data as blob "sha256:a7185479d1426898e8d417574362b30bdaa7a05d8db114432ec13ee724388f03"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Attempting to pull candidate registry.access.redhat.com/redhat/community-operator-index:v4.12 for registry.access.redhat.com/redhat/community-operator-index:v4.12
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.access.redhat.com/redhat/community-operator-index:v4.12"
Trying to pull registry.access.redhat.com/redhat/community-operator-index:v4.12...
DEBU[0000] Copying source image //registry.access.redhat.com/redhat/community-operator-index:v4.12 to destination image [overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.access.redhat.com/redhat/community-operator-index:v4.12
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "registry.access.redhat.com/redhat/community-operator-index:v4.12"
DEBU[0000] No credentials matching registry.access.redhat.com/redhat/community-operator-index found in /run/containers/0/auth.json
DEBU[0000] No credentials matching registry.access.redhat.com/redhat/community-operator-index found in /root/.config/containers/auth.json
DEBU[0000] No credentials matching registry.access.redhat.com/redhat/community-operator-index found in /root/.docker/config.json
DEBU[0000] No credentials matching registry.access.redhat.com/redhat/community-operator-index found in /root/.dockercfg
DEBU[0000] No credentials for registry.access.redhat.com/redhat/community-operator-index found
DEBU[0000]  Lookaside configuration: using "docker" namespace registry.access.redhat.com
DEBU[0000]   Using "sigstore" https://access.redhat.com/webassets/docker/content/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com
DEBU[0000]  Sigstore attachments: using "docker" namespace registry.access.redhat.com
DEBU[0000] GET https://registry.access.redhat.com/v2/
DEBU[0030] Ping https://registry.access.redhat.com/v2/ err Get "https://registry.access.redhat.com/v2/": dial tcp 23.53.42.193:443: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.access.redhat.com/v2/", Err:(*net.OpError)(0xc00056b770)})
DEBU[0030] GET https://registry.access.redhat.com/v1/_ping
DEBU[0060] Ping https://registry.access.redhat.com/v1/_ping err Get "https://registry.access.redhat.com/v1/_ping": dial tcp 23.53.42.210:443: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.access.redhat.com/v1/_ping", Err:(*net.OpError)(0xc0007403c0)})
DEBU[0060] Accessing "registry.access.redhat.com/redhat/community-operator-index:v4.12" failed: pinging container registry registry.access.redhat.com: Get "https://registry.access.redhat.com/v2/": dial tcp 23.53.42.193:443: i/o timeout
WARN[0060] Failed, retrying in 1s ... (1/3). Error: initializing source docker://registry.access.redhat.com/redhat/community-operator-index:v4.12: pinging container registry registry.access.redhat.com: Get "https://registry.access.redhat.com/v2/": dial tcp 23.53.42.193:443: i/o timeout
DEBU[0061] Using registries.d directory /etc/containers/registries.d
DEBU[0061] Trying to access "registry.access.redhat.com/redhat/community-operator-index:v4.12"
DEBU[0061] No credentials matching registry.access.redhat.com/redhat/community-operator-index found in /run/containers/0/auth.json
DEBU[0061] No credentials matching registry.access.redhat.com/redhat/community-operator-index found in /root/.config/containers/auth.json
DEBU[0061] No credentials matching registry.access.redhat.com/redhat/community-operator-index found in /root/.docker/config.json
DEBU[0061] No credentials matching registry.access.redhat.com/redhat/community-operator-index found in /root/.dockercfg
DEBU[0061] No credentials for registry.access.redhat.com/redhat/community-operator-index found
DEBU[0061]  Lookaside configuration: using "docker" namespace registry.access.redhat.com
DEBU[0061]   Using "sigstore" https://access.redhat.com/webassets/docker/content/sigstore
DEBU[0061] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com
DEBU[0061]  Sigstore attachments: using "docker" namespace registry.access.redhat.com
DEBU[0061] GET https://registry.access.redhat.com/v2/
DEBU[0091] Ping https://registry.access.redhat.com/v2/ err Get "https://registry.access.redhat.com/v2/": dial tcp 23.53.42.210:443: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.access.redhat.com/v2/", Err:(*net.OpError)(0xc000769680)})
DEBU[0091] GET https://registry.access.redhat.com/v1/_ping
DEBU[0121] Ping https://registry.access.redhat.com/v1/_ping err Get "https://registry.access.redhat.com/v1/_ping": dial tcp 23.36.162.71:443: i/o timeout (&url.Error{Op:"Get", URL:"https://registry.access.redhat.com/v1/_ping", Err:(*net.OpError)(0xc0007746e0)})
DEBU[0121] Accessing "registry.access.redhat.com/redhat/community-operator-index:v4.12" failed: pinging container registry registry.access.redhat.com: Get "https://registry.access.redhat.com/v2/": dial tcp 23.53.42.210:443: i/o timeout

What's wrong here?

Thanks and greetings,

Boris

Version
4.12.0-0.okd-2023-04-16-041331

How reproducible
100%

[vrutkovs]

Your mirror configuration for registry.access.redhat.com will not be used for tags (like registry.access.redhat.com/redhat/community-operator-index:v4.12), but for manifest digests:

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/redhat"
  mirror-by-digest-only = true

You need to remove / comment the last line

[ObieBent]

Thanks for your reply.
Finally managed this through a machine config and I can successfully pull the image from the specified nodes.
You can close this case 👍

[JaimeMagiera]

A point on communication: The OKD Community has a wide variety of contributors. As such, the greeting "gentleman" is a bit of a misnomer (and a bit outdated). Thanks.

[ObieBent]

Oups sorry for this :(