Ingress operator is degraded due to CanaryChecksRepetitiveFailures when Ingress is behind L7 load-balancer (v4.17)

[manfuin]

Upgrade to 4.17.0-okd-scos.0 stuck for us due to Ingress Operator being degraded.

The error is:

 - lastTransitionTime: "2024-12-16T14:22:25Z"
    message: |-
      One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages: expected canary request body to contain "Healthcheck requested" (x66 over 1h5m0s))
    reason: DegradedConditions
    status: "True"
    type: Degraded
  - lastTransitionTime: "2024-12-16T14:21:21Z"
    message: |-
      Canary route checks for the default ingress controller are failing. Last 1 error messages:
      expected canary request body to contain "Healthcheck requested" (x66 over 1h5m0s)
    reason: CanaryChecksRepetitiveFailures
    status: "False"

Indeed, further troubleshooting shows that querying the canary URL canary-openshift-ingress-canary.apps.<domain> is failing, however the canary Pods are operational and responding correctly when directly queried via e.g. port-forward.

The issue comes down to the fact that our setup uses L7 HAProxy Load-Balancer in front of OKD Ingress Pods that is perfectly fine with Edge/Re-encrypt routes.

The root cause seams the upstream change of Route for canary (https://github.com/openshift/cluster-ingress-operator/blob/master/pkg/manifests/assets/canary/route.yaml) - unconditionally set the type to "passthrough" to address some mTLS issues. It will be great to have a possibility to conditionally specify the route type instead of addressing one corner case while breaking the other.

Mitigation: use L4 HAProxy endpoint in front of Ingress for "canary-openshift-ingress-canary" to let it go with "passthrough". But this makes no sense other that just mitigating upgrade issues - the canary tests are going via completely different L4 path compared to a usual traffic via L7. So the canaries do testing that is not relevant.

Is there actually a way to disable canaries in OKD deployment of Ingress?
I rather disable them if not possible to make them work properly.

[GingerGeek]

Would this mean all passthrough routes would be non-functional in your cluster environment?

[manfuin]

For the default Ingress controller with default settings for wildcard "*.apps" - yes.

We have very few pass-through routes (mostly Kafka) and those are explicitly routed via L4 to work with client certificates auth. But more than 95% of routes are edge routes via L7 HTTP routing on central LB level. We prefer to use L7 by default to have better visibility of source IP address and more sophisticated ACLs in some cases on HTTP level.

As the purpose of canary is to make sure the default path works - I think it should be flexible enough as default paths are vary between installations.

Or at least it makes sense to make canary optional (possible to disable) if favor of other infra means to be used to monitor Routes reachability, instead of blocking the upgrade of OKD with degraded operator.