Assisted Installer failed to fetch OKD image on mirrored repository which is using self-signed certificate

[art-niu]

Hello there,

I am setting up OKD cluster with the guides at https://vrutkovs.eu/posts/okd-disconnected-assisted/#:~:text=Installing%20OKD%20using%20Assisted%20Installer%20in%20a%20disconnected,Run%20Assisted%20Installer%20in%20podman%20...%20More%20items

It failed at "Install Cluster" with below error:

time="2023-10-01T02:42:51Z" level=info msg="Fetching image from OCP release (oc adm release info --image-for=machine-config-operator --insecure=false ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64)" func="github.com/openshift/assisted-service/internal/oc.(*release).getImageFromRelease" file="/assisted-service/internal/oc/release.go:287" pkg=instructions
time="2023-10-01T02:42:51Z" level=warning msg="command 'oc adm release info --image-for=machine-config-operator --insecure=false ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64 --registry-config=/tmp/registry-config541372201' exited with non-zero exit code 1: \nerror: unable to read image ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64: Get \"https://ops-lab.csd.ca/v2/\": x509: certificate signed by unknown authority\n" func=github.com/openshift/assisted-service/internal/oc.execute file="/assisted-service/internal/oc/release.go:404" pkg=instructions
time="2023-10-01T02:42:51Z" level=error msg="failed to get machine-config-operator image from release image ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64" func="github.com/openshift/assisted-service/internal/oc.(*release).getImageByName" file="/assisted-service/internal/oc/release.go:131" error="command 'oc adm release info --image-for=machine-config-operator --insecure=false ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64 --registry-config=/tmp/registry-config541372201' exited with non-zero exit code 1: \nerror: unable to read image ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64: Get \"https://ops-lab.csd.ca/v2/\": x509: certificate signed by unknown authority\n" pkg=instructions

Where is the trust store which is used by the installer? How can it be loaded to the trust store before the installation started.

Thanks,

Arthur

[vrutkovs]

Get "https://ops-lab.csd.ca/v2/\": x509: certificate signed by unknown authority

See https://github.com/openshift/assisted-service/blob/master/deploy/podman/README.md#tls-certificates

[art-niu]

The instructions is to set Assisted Installer Service with TLS. The problem which I ran into is:
The Assisted Installer service failed to fetch the mirrored docker registry because the docker registry service using self-signed certificate.
Is there a way to ignore the self-signed when Assisted Installer Service fetch the docker image?

time="2023-10-02T20:02:17Z" level=warning msg="command 'oc adm release info --image-for=machine-config-operator --insecure=false ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64 --registry-config=/tmp/registry-config2349277634' exited with non-zero exit code 1: \nerror: unable to read image ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64: Get \"https://ops-lab.csd.ca/v2/\": x509: certificate signed by unknown authority\n" func=github.com/openshift/assisted-service/internal/oc.execute file="/assisted-service/internal/oc/release.go:404" pkg=instructions

[art-niu]

@vrutkovs is there a way to set insecure=true to read image ops-lab.csd.ca/openshift/release-images:4.13.0-0.okd-2023-09-03-082426-x86_64?

[vrutkovs]

No. You need to add registry's CA in CA bundle of the assisted service container

[art-niu]

@vrutkovs I got below error.

starting container 874719bfb5306ee3aedaca5d1475d1d8ebb58dcd156ec6ef5589fd8bf7504a07: runc: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/containers/storage/volumes/config/_data/tls-ca-bundle.pem" to rootfs at "/tmp": mount /var/lib/containers/storage/volumes/config/_data/tls-ca-bundle.pem:/tmp (via /proc/self/fd/6), flags: 0x5006: not a directory: OCI runtime error

Then I tried to run container with command line, and got same problem:

podman run -d --name my-container \
  -v /var/lib/containers/storage/volumes/config/_data/tls-ca-bundle.pem:/etc/pki/ca-trust/extracted/pem \
  ops-lab.csd.toronto.ca/edge-infrastructure/assisted-service:latest

Could you please advise?

Thanks,
Arthur

[vrutkovs]

/var/lib/containers/storage/volumes/config/_data/tls-ca-bundle.pem

This path is probably on an overlay FS. Is this a separate podman volume?
Check if it works when CA bundle is at some other location, outside of /var/lib/containers/storage