In cluster DNS issue

[peishuli]

Hello there!

We are evaluating solutions for cross-(k8s)cluster LB and recently came across k8gb. I adopted the Azure DNS tutorial by swapping out the two AKS clusters with two RKE2 clusters (although running on Azure VMs to simulate on-prem clusters). I managed to make it work but with some limitations. The biggest issue we are still facing is that whenever we expose k8gb-coredns (either via LB or enabling udp.53 in ingress-nginx chart), coredns will take over DNS name resolving - i.e., new pods would fall pulling images from anywhere due to name resolving issue). The workaround it to temporarily turn off coredns LB svc so that you can deploy something and then turn LB on coredns back on (and set imagePullPolicy to IfNotPresent so when you scale the deployment down to 0 replica and back to 1 again, it won't trying to repull the image).

Below are my configurations:

#values file contents for ingress-nginx:

controller:
  kind: DaemonSet
  service:
    annotations:
      [service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path](http://service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path): /healthz

#values file contents for k8gb (of one region):

k8gb:
  dnsZones:
    - loadBalancedZone: "[k8gb.newrezk8snonprod.com](http://k8gb.newrezk8snonprod.com/)"
      parentZone: "[newrezk8snonprod.com](http://newrezk8snonprod.com/)"
      dnsZoneNegTTL: 300 # -- Negative TTL for SOA record
  edgeDNSServers:
    - "1.1.1.1"
    - "8.8.8.8"
  clusterGeoTag: "ga"
  extGslbClustersGeoTags: "tx"
  reconcileRequeueSeconds: 10
externaldns:
  interval: "10s"
azuredns:
  enabled: true
  createAuthSecret:
    enabled: true
    tenantId: ...
    subscriptionId: ...
    resourceGroup: ...
    aadClientId: ...
    aadClientSecret: ...
coredns:
  serviceType: "LoadBalancer"
istio:
  enabled: false

Thanks,
Peishu

[ytsarev]

Can you please format yaml for clarify?

whenever we expose k8gb-coredns (either via LB or enabling udp.53 in ingress-nginx chart), coredns will take over DNS name resolving

It should not happen in the standard setup. Can you please clarify why the cluster is starting to use k8gb-coredns for resolution? What in your DNS setup points to k8gb-coredns service?

[peishuli]

Sorry about the formatting and they are fixed.

The issue is - when k8gb is deployed with its embedded k8gb-coredns exposed (e.g., as LB servicetype), DNS for any pods in the cluster will be taken over by it.

For instance, once k8gb-coredns is exposed, lookup google.com will fail but nslookup nginx.k8gb.newrezk8snonprod.com works. This behavior exists even at the node level (I was able to verify it by ssh into the nodes and run nslookup commands).

[abaguas]

Did you change any configuration of your nodes to use k8gb's coreDNS?

Your pods are supposed to use coredns on the kube-system namespace, not k8gb's coredns

[peishuli]

Hi @abaguas, I did NOT. The clusters running RKE2 which comes with it's one coredns that is running inside kube-system ns:

kubectl -n kube-system get po --kubeconfig=/mnt/c/temp/rke-tx-poc.yaml | grep coredns
rke2-coredns-rke2-coredns-65dc69968-dmkvq              1/1     Running     0             18h
rke2-coredns-rke2-coredns-65dc69968-w67gg              1/1     Running     0             18h
rke2-coredns-rke2-coredns-autoscaler-68d5f76f7-bzjg7   1/1     Running     0             18h

We also enabled Service Load Balancer on RKE2.

In these POC clusters, we only have two work nodes each.

As I mentioned earlier, when k8gb-coredns is exposed (either via LB or via ingress controller's udp.53 configuration), it affects the DNS on the nodes as well, not just inside the containers.

[peishuli]

Question - what does k8gb.edgeDNSServers do? I added 1.1.1.1 and 8.8.8.8 to the list. I guess it's used for resolving external domains (e.g., docker.io, etc.), isn't it? If so, maybe it's not be used?

I've noticed in your Azure DNS sample, it was parentZoneDNSServers. I had to change it to edgeDNSServers otherwise the helm chart won't be deployed due to schema violation.

[abaguas]

The edge dns servers are the dns servers used by k8gb to resolve dns records.
The primary use is to discover coredns installations in other clusters. So it needs to be able to resolve records hosted in the "parentZone".

If your loadBalancer services have a hostname, then I think it would also query the edge dns servers.

If your zone is publicly accessible then any public dns servers would work, such as 1.1.1.1 and 8.8.8.8.
Using Azure's dns servers is also a good option: 168.63 129.16. This way you can also resolve DNS records only available in your vnet

[abaguas]

I've noticed in your Azure DNS sample, it was parentZoneDNSServers. I had to change it to edgeDNSServers otherwise the helm chart won't be deployed due to schema violation.

Thanks, we will fix this

[peishuli]

The edge dns servers are the dns servers used by k8gb to resolve dns records. The primary use is to discover coredns installations in other clusters. So it needs to be able to resolve records hosted in the "parentZone".

If your loadBalancer services have a hostname, then I think it would also query the edge dns servers.

If your zone is publicly accessible then any public dns servers would work, such as 1.1.1.1 and 8.8.8.8. Using Azure's dns servers is also a good option: 168.63 129.16. This way you can also resolve DNS records only available in your vnet

Thanks for the explanation, that makes perfect sense!

Any idea why k8gb-coredns in my clusters would "hijack" the entire cluster's domain name resolution including the nodes?

[peishuli]

Hello @abaguas and @ytsarev,

After some digging (and googling :), I was able to find the root cause of the problem. Quoted below from Google's AI Overview:

"When CoreDNS is exposed as a LoadBalancer service in Kubernetes on an Ubuntu node, it can sometimes interfere with the node's own DNS resolution. This issue often arises from the interaction between CoreDNS, the kubelet's DNS configuration, and the node's local DNS resolver (e.g., systemd-resolved on Ubuntu).

Potential Causes and Solutions:
systemd-resolved Stub File and Forwarding Loops:

• Issue: Ubuntu's systemd-resolved creates a stub resolv.conf file, which can lead to a fatal forwarding loop if CoreDNS attempts to resolve names through this stub file, ultimately pointing back to itself or causing recursive lookups.

• Solution: Configure kubelet to use the correct resolv.conf file by setting the --resolv-conf flag in its configuration. For systemd-resolved, this typically points to /run/systemd/resolve/resolv.conf."

It correctly pointed the finger to Ubuntu linux which is what I used for our Azure VMs. I simply switched from Ubuntu image to a SUSE Linux image in our Terraform module and recreated the cluster, set up k8gb and voilà, the problem was gone!

When I get a chance, I may try their proposed solution (i.e., configuring kubelet's pointer to the correct resolv.conf file in Ubuntu) so that people can choose Ubuntu if they want to.

Another thing that tripped me up earlier was my misunderstanding the relationship between clusterGeoTag and extGslbClustersGeoTags. I attributed that misunderstanding to the default values.yaml file in the repo where it listed clusterGeoTag: "eu" and extGslbClustersGeoTags: "eu,us". This configuration will cause the external-dns pod to fail on the NS record update (in the edge DNS server) because it would try to write the eu-* record twice and end up with none of them be written there at all. Based on this observation, here is what I understand now: In extGslbClustersGeoTags you put everything geo tags EXCEPT the very one you're deploy. Please remove the "us" from extGslbClustersGeoTags in the default values.yaml file or change it to something like "eu,cn".

Great product! I believe some clarity in the documentation and up-to-date samples will go a long way in improving its adaption.

I am ready to demonstrate it to our management team next week for a possible buy-in. Other options we are considering include Linkerd's mutli-cluster failover and F5 BIG IP GSLB. IMHO, k8gb beats both of them.

Thanks a lot for your promptly response and excellent explanation on how k8gb works!

[abaguas]

We are glad the problem is now understood.
Thank you for the feedback, that is very valuable. We will fix the points you raised.

If you have any further questions let us know

[peishuli]

Thank you @abaguas.

I tried to add a third cluster/region but observed some very bizarre behaviors. It could be some misconfigurations somewhere. Do you have samples with 3 or more regions?

Additionally I want to test Weight(ed) Round Robin set up but wasn't able to find any samples in the doc on how to do it. Ingress annotations seems only allow failover and roundRobin options. For weighted round robin, do I have to use explicitly create a Gslb CR and configure it there? What about GeoIP examples? I could not find it in the doc either.