DNS broken due to ndots

[reavessm]

Describe the bug
DNS lookups seem to fail unless options ndots:5 is changed to at most options ndots:2 in /etc/resolv.conf. I noticed weird issues with some pods getting an error saying x509: certificate is valid for *.apps.openshift.<internalDomain> not proxy.golang.org when running a golang container for example. I noticed with the ubuntu and fedora containers, running an apt/dnf update would also fail because the pkg repositories were resolving to my cluster domain rather where they actually should. Removing options ndots:5 fixed all my problems, but I don't know how to set this cluster-wide, nor do I know if there are any security implications.

FWIW, I was on 4.7.13, and everything seemed to be fine, but upgrading to 4.7.18 seemed to break, and neither 4.7.19 nor 4.7.21 seemed to fix it.

Version
Client Version: 4.6.6
Server Version: 4.7.21
Kubernetes Version: v1.20.0+558d959

[reavessm]

Update, After rebooting the cluster, the affected pods were obviously recreated, and therefore I lost my changes to the /etc/resolv.conf. Now I can't attach to the pods at all. I've tried connecting through the terminal tab in the web-console, I've tried using oc attach deployment/name, and oc rsh pod/name-number but they all fail. The last two at least give me message saying Error from server: error dialing backend: x509: certificate is valid for 192.168.0.192, not 192.168.0.200 which makes me think this is related.

So now I can't connect to the pods to make any changes, so they're basically unusable at this point. Any information would be helpful

[vrutkovs]

All versions mentioned are OCP, please contact Red Hat Support instead