Service Type LoadBalancer not accessible from inside the cluster pods with OVN-Kubernetes

[m-yosefpor]

Describe the bug

Services of type LoadBalancer are not accessible from inside the pods of the cluster, although they are accessible from cluster nodes and pods with hostNetworking, or outside of the cluster.

It encounters with an immediate connection refused:

$ curl  http://172.21.60.1:8080
curl: (7) Failed connect to 172.21.60.1:8080; Connection refused

Version

4.7.0-0.okd-2021-07-03-190901
UPI, Openstack
OVN-Kubernetes

How reproducible
Please specify how often the issue is reproducible: 100%

Here is a service type loadbalancer (external IP and BGP advertisement have been done with metallb):

$ oc get svc health-be-lb
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
health-be-lb   LoadBalancer   172.30.134.172   172.21.60.1   8080:32003/TCP   111d

curling it from outside of the cluster, or even from each nodes of the cluster is working OK:

$ curl -I http://172.21.60.1:8080
HTTP/1.1 200 OK

Even from pods with hostNetworking it is working OK:

$ oc rsh -n openshift-ingress router-default-9764568c7-f2jnz
sh-4.4$ curl -I  http://172.21.60.1:8080
HTTP/1.1 200 OK

Now we try curling it from inside a cluster pod without hostNetworking (e.g. prometheus). We will face immediate connection refused error.

$ oc rsh -n openshift-monitoring -c prometheus prometheus-k8s-0
sh-4.4$ curl http://172.21.60.1:8080
curl: (7) Failed to connect to 172.21.60.1 port 8080: Connection refused

Here is a packet capture result on the host where prometheus is, which shows it immediately receives a Return:

$ tcpdump -vvvniany host 172.21.60.1
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
15:46:41.571556 f7cd1ba8b56c643 P   IP (tos 0x0, ttl 64, id 64305, offset 0, flags [DF], proto TCP (6), length 60)
    10.130.8.21.54546 > 172.21.60.1.8080: Flags [S], cksum 0xfadb (incorrect -> 0x32d2), seq 1690399808, win 65280, options [mss 1360,sackOK,TS val 1326907176 ecr 0,nop,wscale 7], length 0
15:46:41.573169 f7cd1ba8b56c643 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    172.21.60.1.8080 > 10.130.8.21.54546: Flags [R.], cksum 0xe77d (correct), seq 0, ack 1690399809, win 0, length 0

Additional Info:
No network policy has been set.
Accessing NodePort services are OK from both pods with hostNetworking and non-hostNetworking.
The same setup is working with okd v3.11 with openshift-sdn.

Log bundle
Not applicable.

[vrutkovs]

Is this OKD-specific or reproducible on OCP too?

[m-yosefpor]

I don't know, I don't have an OCP cluster to test this on it. However most probably it is related to OVN-k8s component which is used in both platforms.

[vrutkovs]

Lets file a bugzilla just to be sure

[m-yosefpor]

https://bugzilla.redhat.com/show_bug.cgi?id=1984690