Workers load balancer is down on GCP - fresh installation

[llomgui]

Describe the bug

I tried many times to install OKD 4.7 on GCP. But it doesn't work.
Each time, masters load balancer is up, but workers load balancer is down.

Version

Fedora coreos: projects/fedora-coreos-cloud/global/images/fedora-coreos-33-20210217-3-0-gcp-x86-64

I tested with the latest: 4.7.0-0.okd-2021-06-19-191547
And 4.7.0-0.okd-2021-04-24-103438

The difference between them are GCP images (gcp-machine-controllers, gcp-pd-csi-driver, gcp-pd-csi-driver-operator)

How reproducible

You can try to install a cluster with the versions above.

Log bundle

Tcpdump on workers (35.191.0.0/16 is one of healthcheck ip ranges) :

⬢[root@toolbox ~]# tcpdump net 35.191.0.0/16
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:24:57.870853 IP 93-9-191-35.1e100.net.56282 > 10.69.18.29.31947: Flags [S], seq 2878974984, win 65535, options [mss 1420,sackOK,TS val 3319591662 ecr 0,nop,wscale 8], length 0
08:24:57.870886 IP 93-9-191-35.1e100.net.56282 > 10.69.18.29.31947: Flags [S], seq 2878974984, win 65535, options [mss 1420,sackOK,TS val 3319591662 ecr 0,nop,wscale 8], length 0
08:24:58.970867 IP 91-9-191-35.1e100.net.49926 > 10.69.18.29.31947: Flags [S], seq 2200351049, win 65535, options [mss 1420,sackOK,TS val 1361109183 ecr 0,nop,wscale 8], length 0
08:24:58.970900 IP 91-9-191-35.1e100.net.49926 > 10.69.18.29.31947: Flags [S], seq 2200351049, win 65535, options [mss 1420,sackOK,TS val 1361109183 ecr 0,nop,wscale 8], length 0
08:24:59.231184 IP 89-9-191-35.1e100.net.51776 > 10.69.18.29.31947: Flags [S], seq 2492519718, win 65535, options [mss 1420,sackOK,TS val 1719082114 ecr 0,nop,wscale 8], length 0
08:24:59.231205 IP 89-9-191-35.1e100.net.51776 > 10.69.18.29.31947: Flags [S], seq 2492519718, win 65535, options [mss 1420,sackOK,TS val 1719082114 ecr 0,nop,wscale 8], length 0
08:25:05.871865 IP 93-9-191-35.1e100.net.46610 > 10.69.18.29.31947: Flags [S], seq 3336617057, win 65535, options [mss 1420,sackOK,TS val 3319599663 ecr 0,nop,wscale 8], length 0
08:25:05.871911 IP 93-9-191-35.1e100.net.46610 > 10.69.18.29.31947: Flags [S], seq 3336617057, win 65535, options [mss 1420,sackOK,TS val 3319599663 ecr 0,nop,wscale 8], length 0
08:25:06.972166 IP 91-9-191-35.1e100.net.42364 > 10.69.18.29.31947: Flags [S], seq 2659682155, win 65535, options [mss 1420,sackOK,TS val 1361117184 ecr 0,nop,wscale 8], length 0
08:25:06.972213 IP 91-9-191-35.1e100.net.42364 > 10.69.18.29.31947: Flags [S], seq 2659682155, win 65535, options [mss 1420,sackOK,TS val 1361117184 ecr 0,nop,wscale 8], length 0
08:25:07.232678 IP 89-9-191-35.1e100.net.52286 > 10.69.18.29.31947: Flags [S], seq 2445915522, win 65535, options [mss 1420,sackOK,TS val 1719090116 ecr 0,nop,wscale 8], length 0
08:25:07.232704 IP 89-9-191-35.1e100.net.52286 > 10.69.18.29.31947: Flags [S], seq 2445915522, win 65535, options [mss 1420,sackOK,TS val 1719090116 ecr 0,nop,wscale 8], length 0

As you can see, I only get SYN, it never answers.

Same command on one of masters:

⬢[root@toolbox ~]# tcpdump net 35.191.0.0/16
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:27:51.264928 IP 27-0-191-35.1e100.net.60690 > 10.69.18.24.sun-sr-https: Flags [S], seq 2753378919, win 65535, options [mss 1420,sackOK,TS val 3835801912 ecr 0,nop,wscale 8], length 0
08:27:51.264996 IP 10.69.18.24.sun-sr-https > 27-0-191-35.1e100.net.60690: Flags [S.], seq 722652261, ack 2753378920, win 64768, options [mss 1420,sackOK,TS val 3241567184 ecr 3835801912,nop,wscale 7], length 0
08:27:51.265121 IP 27-0-191-35.1e100.net.60690 > 10.69.18.24.sun-sr-https: Flags [.], ack 1, win 256, options [nop,nop,TS val 3835801912 ecr 3241567184], length 0
08:27:51.265230 IP 27-0-191-35.1e100.net.60690 > 10.69.18.24.sun-sr-https: Flags [P.], seq 1:255, ack 1, win 256, options [nop,nop,TS val 3835801913 ecr 3241567184], length 254
08:27:51.265243 IP 10.69.18.24.sun-sr-https > 27-0-191-35.1e100.net.60690: Flags [.], ack 255, win 505, options [nop,nop,TS val 3241567184 ecr 3835801913], length 0
08:27:51.268662 IP 10.69.18.24.sun-sr-https > 27-0-191-35.1e100.net.60690: Flags [P.], seq 1:3021, ack 255, win 505, options [nop,nop,TS val 3241567188 ecr 3835801913], length 3020
08:27:51.268856 IP 27-0-191-35.1e100.net.60690 > 10.69.18.24.sun-sr-https: Flags [.], ack 3021, win 280, options [nop,nop,TS val 3835801916 ecr 3241567188], length 0
08:27:51.269035 IP 27-0-191-35.1e100.net.60690 > 10.69.18.24.sun-sr-https: Flags [P.], seq 255:327, ack 3021, win 280, options [nop,nop,TS val 3835801916 ecr 3241567188], length 72
08:27:51.269050 IP 10.69.18.24.sun-sr-https > 27-0-191-35.1e100.net.60690: Flags [.], ack 327, win 505, options [nop,nop,TS val 3241567188 ecr 3835801916], length 0
08:27:51.269150 IP 27-0-191-35.1e100.net.60690 > 10.69.18.24.sun-sr-https: Flags [P.], seq 327:442, ack 3021, win 280, options [nop,nop,TS val 3835801916 ecr 3241567188], length 115
08:27:51.269161 IP 10.69.18.24.sun-sr-https > 27-0-191-35.1e100.net.60690: Flags [.], ack 442, win 505, options [nop,nop,TS val 3241567188 ecr 3835801916], length 0

It works as expected.

[LucaPrete]

Just to echo what @llomgui is saying,

As you can see, workers seem to only receive SYN packets from the health checks, but they don't generate outgoing packets, letting us think the machine is dropping those packets.

[vrutkovs]

Please attach log bundle

[llomgui]

Yes it is.

The first LB is generated by OKD during workers installation.

But healthcheck doesn't validate the nodes.

Healthcheck's configuration:

[llomgui]

@vrutkovs Same issue with the version 4.7.0-0.okd-2021-07-03-190901.

On master nodes in journalctl:

Jul 04 13:09:02 okd-ci-dm9zv-master-a.okd-ci.euwe6.domain.xyz bash[754]: Processing route for NIC 0/42:01:0a:45:12:22 for 10.69.18.33
Jul 04 13:09:02 okd-ci-dm9zv-master-a.okd-ci.euwe6.domain.xyz bash[754]: Processing route for NIC 0/42:01:0a:45:12:22 for 10.69.18.38
Jul 04 13:09:02 okd-ci-dm9zv-master-a.okd-ci.euwe6.domain.xyz bash[754]: ensuring rule for 10.69.18.33 for external clients
Jul 04 13:09:02 okd-ci-dm9zv-master-a.okd-ci.euwe6.domain.xyz bash[754]: ensuring rule for 10.69.18.33 for internal clients
Jul 04 13:09:02 okd-ci-dm9zv-master-a.okd-ci.euwe6.domain.xyz bash[754]: ensuring rule for 10.69.18.38 for external clients
Jul 04 13:09:02 okd-ci-dm9zv-master-a.okd-ci.euwe6.domain.xyz bash[754]: ensuring rule for 10.69.18.38 for internal clients
Jul 04 13:09:02 okd-ci-dm9zv-master-a.okd-ci.euwe6.domain.xyz bash[754]: done applying vip rules

We don't have this on worker nodes. Where does it come from?

[vrutkovs]

The first LB is generated by OKD

I don't see the code which does that in the installer. Is it the censored LB on your screenshot? What's its purpose?

[llomgui]

Yes this one.
Its purpose is to handle resquests from 80,443 and redirect them to worker nodes (router pods are running on them).

[vrutkovs]

ts purpose is to handle resquests from 80,443 and redirect them to worker nodes (router pods are running on them)

Oh, so ingress creates an LB which works in a happy path, but it won't be able to exclude unhealthy VMs (unlike api/api-int)? That's pretty serious.

[llomgui]

@vrutkovs Running this script https://github.com/openshift/machine-config-operator/blob/master/templates/master/00-master/gcp/files/opt-libexec-openshift-gcp-routes-sh.yaml on worker nodes works like a charm.

Is it possible to include it also for worker nodes?

[vrutkovs]

Resolved in 4.8 - openshift/okd-machine-os#132

[llomgui]

Do you have a release date for OKD 4.8?

Is it possible to have a backport for OKD 4.7?

[vrutkovs]

Do you have a release date for OKD 4.8?

Not yet, depends on test results. Hopefully it would be ready in few weeks.

Is it possible to have a backport for OKD 4.7?

Yes, but that would take some time - we'd like to have this script synced between OCP and OKD, so we need a backport to land in OCP first

[vrutkovs]

Lets track this in #743. One could work this around by creating an MC which writes the script / systemd service on the nodes meanwhile.

Thanks for bearing with me on this :)

[llomgui]

Create 99-worker-gcp-routes-sh.bu with the following:

variant: openshift
version: 4.8.0
metadata:
  name: 99-worker-gcp-routes-sh
  labels:
    machineconfiguration.openshift.io/role: worker
storage:
  files:
    - path: /opt/libexec/openshift-gcp-routes-sh
      mode: 0755
      contents:
        inline: |
          #!/bin/bash
          # Update iptables rules based on google cloud load balancer VIPS
          #
          # This is needed because the GCP L3 load balancer doesn't actually do DNAT;
          # the destination IP address is still the VIP. Normally, there is an agent that
          # adds the vip to the local routing table, tricking the kernel in to thinking
          # it's a local IP and allowing processes doing an accept(0.0.0.0) to receive
          # the packets. Clever.
          #
          # We don't do that. Instead, we DNAT with conntrack. This is so we don't break
          # existing connections when the vip is removed. This is useful for draining
          # connections - take ourselves out of the vip, but service existing conns.
          #
          # Additionally, clients can write a file to /run/cloud-routes/$IP.down to force
          # a VIP as down. This is useful for graceful shutdown / upgrade.
          #
          # ~cdc~
          set -e
          # the list of load balancer IPs that are assigned to this node
          declare -A vips
          curler() {
            curl --silent -L -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/${1}"
          }
          CHAIN_NAME="gcp-vips"
          RUN_DIR="/run/cloud-routes"
          # Create a chan if it doesn't exist
          ensure_chain() {
              local table="${1}"
              local chain="${2}"
              if ! iptables -w -t "${table}" -S "${chain}" &> /dev/null ; then
                  iptables -w -t "${table}" -N "${chain}";
              fi;
          }
          ensure_rule() {
              local table="${1}"
              local chain="${2}"
              shift 2
              if ! iptables -w -t "${table}" -C "${chain}" "$@" &> /dev/null; then
                  iptables -w -t "${table}" -A "${chain}" "$@"
              fi
          }
          # set the chain, ensure entry rules, ensure ESTABLISHED rule
          initialize() {
              ensure_chain nat "${CHAIN_NAME}"
              ensure_chain nat "${CHAIN_NAME}-local"
              ensure_rule nat PREROUTING -m comment --comment 'gcp LB vip DNAT' -j ${CHAIN_NAME}
              ensure_rule nat OUTPUT -m comment --comment 'gcp LB vip DNAT for local clients' -j ${CHAIN_NAME}-local
              # Need this so that existing flows (with an entry in conntrack) continue to be
              # balanced, even if the DNAT entry is removed
              ensure_rule filter INPUT -m comment --comment 'gcp LB vip existing' -m addrtype ! --dst-type LOCAL -m state --state ESTABLISHED,RELATED -j ACCEPT
              # bz1925698: GCP LBs can create stale entries causing apiservers disruption
              # The GCP LB Health Check polls continuously the LB VIP assigned to the VM
              # but, if the LB VIP is down, we are not handling the VIP traffic and
              # the traffic can hairpin, leaving stale conntrack entries on the host.
              # xref: https://bugzilla.redhat.com/show_bug.cgi?id=1925698#c29
              # Deleting conntrack entries solves the problem, but it also affects other connections
              # directed to the LB vips, causing unexpected networks disruptions.
              # The solution is to not FORWARD GCP HealthCheckers traffic, because that traffic 
              # is only directed to the host, and it must go to the INPUT chain.
              # xref: https://bugzilla.redhat.com/show_bug.cgi?id=1930457#c8
              # The HealthCheck origin ip-ranges are documented: 130.211.0.0/22 and 35.191.0.0/16
              # xref: https://cloud.google.com/load-balancing/docs/health-check-concepts#ip-ranges
              ensure_rule filter FORWARD -m comment --comment 'gcp HealthCheck traffic' -s 35.191.0.0/16 -j DROP
              ensure_rule filter FORWARD -m comment --comment 'gcp HealthCheck traffic' -s 130.211.0.0/22 -j DROP
              mkdir -p "${RUN_DIR}"
          }
          remove_stale() {
              ## find extra iptables rules
              for ipt_vip in $(iptables -w -t nat -S "${CHAIN_NAME}" | awk '$4{print $4}' | awk -F/ '{print $1}'); do
                  if [[ -z "${vips[${ipt_vip}]}" ]]; then
                      echo removing stale vip "${ipt_vip}" for external clients
                      iptables -w -t nat -D "${CHAIN_NAME}" --dst "${ipt_vip}" -j REDIRECT
                  fi
              done
              for ipt_vip in $(iptables -w -t nat -S "${CHAIN_NAME}-local" | awk '$4{print $4}' | awk -F/ '{print $1}'); do
                  if [[ -z "${vips[${ipt_vip}]}" ]] || [[ "${vips[${ipt_vip}]}" = down ]]; then
                      echo removing stale vip "${ipt_vip}" for local clients
                      iptables -w -t nat -D "${CHAIN_NAME}-local" --dst "${ipt_vip}" -j REDIRECT
                  fi
              done
          }
          add_rules() {
              for vip in "${!vips[@]}"; do
                  echo "ensuring rule for ${vip} for external clients"
                  ensure_rule nat "${CHAIN_NAME}" --dst "${vip}" -j REDIRECT
                  if [[ "${vips[${vip}]}" != down ]]; then
                      echo "ensuring rule for ${vip} for internal clients"
                      ensure_rule nat "${CHAIN_NAME}-local" --dst "${vip}" -j REDIRECT
                  fi
              done
          }
          clear_rules() {
              iptables -t nat -F "${CHAIN_NAME}" || true
              iptables -t nat -F "${CHAIN_NAME}-local" || true
          }
          # out paramater: vips
          list_lb_ips() {
              for k in "${!vips[@]}"; do
                  unset vips["${k}"]
              done
              local net_path="network-interfaces/"
              for vif in $(curler ${net_path}); do
                  local hw_addr; hw_addr=$(curler "${net_path}${vif}mac")
                  local fwip_path; fwip_path="${net_path}${vif}forwarded-ips/"
                  for level in $(curler "${fwip_path}"); do
                      for fwip in $(curler "${fwip_path}${level}"); do
                          if [[ -e "${RUN_DIR}/${fwip}.down" ]]; then
                              echo "${fwip} is manually marked as down, skipping for internal clients..."
                              vips[${fwip}]="down"
                          else
                              echo "Processing route for NIC ${vif}${hw_addr} for ${fwip}"
                              vips[${fwip}]="${fwip}"
                          fi
                      done
                  done
              done
          }
          sleep_or_watch() {
              if hash inotifywait &> /dev/null; then
                  inotifywait -t 30 -r "${RUN_DIR}" &> /dev/null || true
              else
                  # no inotify, need to manually poll
                  for i in {0..5}; do
                      for vip in "${!vips[@]}"; do
                          if [[ "${vips[${vip}]}" != down ]] && [[ -e "${RUN_DIR}/${vip}.down" ]]; then
                              echo "new downfile detected"
                              break 2
                          elif [[ "${vips[${vip}]}" = down ]] && ! [[ -e "${RUN_DIR}/${vip}.down" ]]; then
                              echo "downfile disappeared"
                              break 2
                          fi
                      done
                      sleep 1 # keep this small enough to not make gcp-routes slower than LBs on recovery
                  done
              fi
          }
          case "$1" in
            start)
              initialize
              while :; do
                list_lb_ips
                remove_stale
                add_rules
                echo "done applying vip rules"
                sleep_or_watch
              done
              ;;
            cleanup)
              clear_rules
              ;;
            *)
              echo $"Usage: $0 {start|cleanup}"
              exit 1
          esac

Execute butane command to generate yaml file:
butane 99-worker-gcp-routes-sh.bu -o 99-worker-gcp-routes-sh.yaml

Create 99-worker-gcp-routes-service.bu with the following:

variant: openshift
version: 4.8.0
metadata:
  name: 99-worker-gcp-routes-service
  labels:
    machineconfiguration.openshift.io/role: worker
systemd:
  units:
    - name: openshift-gcp-routes.service
      enabled: true
      contents: |
        [Unit]
        Description=Update GCP routes for forwarded IPs.
        ConditionKernelCommandLine=|ignition.platform.id=gce
        ConditionKernelCommandLine=|ignition.platform.id=gcp
        Before=network-online.target
        [Service]
        Type=simple
        ExecStart=/bin/bash /opt/libexec/openshift-gcp-routes.sh start
        ExecStopPost=/bin/bash /opt/libexec/openshift-gcp-routes.sh cleanup
        User=root
        RestartSec=30
        Restart=always
        [Install]
        WantedBy=multi-user.target
        # Ensure that network-online.target will not complete until the node has working external LBs.
        RequiredBy=network-online.target

Execute butane command to generate yaml file:
butane 99-worker-gcp-routes-service.bu -o 99-worker-gcp-routes-service.yaml