Baremetal UPI with a proxy is not bootstrapping

[vdveldet]

I'm trying to install an UPI baremetal environment on a restricted network that has proxy access to the internet.

The bootstrap is located on a KVM server and booted from a QCOW2 image and the masters are Bare metal so booted from an ISO and ignition files are applied during installation.

The bootstrap does not finish, looks like the master servers are not loading the network configuration. All servers are accessible from a bastion and updates are applied to the Fedora Core OS

[vdveldet]

Log bundle attached
log-bundle-20230919201008.tar.gz

[rvanderp3]

I see that the control plane nodes ignited and that the node bootstrapper CSRs were approved. I don't see any additional outstanding CSRs from looking at the log bundle but it looks like based on the errors in the control plane node kubelet logs[0] that there are more CSRs or the certs aren't being issued to the control plane nodes[1].

0 -

Sep 19 20:10:10 master01.ocp4.shared.local kubenswrapper[2024]: I0919 20:10:10.758558    2024 csi_plugin.go:913] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "master01.ocp4.shared.local" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope

1 -

Sep 19 20:05:52 master01.ocp4.shared.local kubenswrapper[2024]: I0919 20:05:52.499251    2024 csr.go:261] certificate signing request csr-k9sp5 is approved, waiting to be issued

[vdveldet]

Can this be a proxy configuration issue ?

[rvanderp3]

It's hard to say from sure from the logs. The control plane nodes were able to request a CSR so I'm not convinced it's a proxy issue myself. Additionally, the api-int endpoint is included with the noProxy:

noProxy: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,100.98.254.0/26,100.98.254.128/25,100.98.255.0/24,127.0.0.1,129.39.139.192/26,129.39.7.0/24,172.30.0.0/16,195.118.243.64/27,62.185.95.128/26,62.185.95.96/27,api-int.ocp4.shared.local,localhost

[vdveldet]

thinking out loud ....

cri-o does not pickup the configuration on the master.

[core@master01 ~]$ sudo crictl info
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": false,
        "reason": "NetworkPluginNotReady",
        "message": "Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
      }
    ]
  }
}

Assuming that the transport is working it is perhapse not generated on the bootstrap server.