BUG: After replacing default ingress certificate authentication operator degraded

[guliaka]

BUG: After replacing default ingress certificate authentication operator degraded

Describe the bug
After replacing default ingress certificate authentication operator degraded with error:

RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.kube-prd.getty.edu] -n openshift-authentication: certificate could not validate route hostname v4-0-config-system-custom-router-certs.apps.kube-prd.getty.edu: x509: certificate signed by unknown authority  
 

Version
4.9.0-0.okd-2022-01-29-035536 which is upgrade of 4.9.0-0.okd-2022-01-14-230113, IPI on vSphere 6.7.0.50000 with ESXi 6.7.0, 17700523

How reproducible
Tried 2 times, both failed
Log bundle
openshift-authentication.zip

Full must-gather: https://drive.google.com/file/d/1JfEzQKFcSU0K63HMnOvLc62dhJY36d1f/view?usp=sharing

[vrutkovs]

Please attach log bundle

[guliaka]

Would you advise me please how to get logs which may help you??

[vrutkovs]

An unabridged must-gather archive would be helpful. From https://github.com/openshift/okd/files/8052780/openshift-authentication.zip I see that v4-0-config-system-router-certs secret has some invalid string set instead of a certificate - but no clue who or what set it

[guliaka]

Full must-gather is here: https://drive.google.com/file/d/1JfEzQKFcSU0K63HMnOvLc62dhJY36d1f/view?usp=sharing

[vrutkovs]

In namespaces/openshift-ingress-operator/operator.openshift.io/ingresscontrollers/default.yaml a cert from the secret is used:

apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
..
  name: default
  namespace: openshift-ingress-operator
..
spec:
  ...
  defaultCertificate:
    name: incommon-wc-ingress

It looks malformed:

$ cat namespaces/openshift-ingress/core/secrets.yaml | grep -C4 "incommon-wc-ingress" 
    tls.key: MTcwNCBieXRlcyBsb25n
  kind: Secret
  metadata:
    creationTimestamp: "2022-02-10T23:46:46Z"
    name: incommon-wc-ingress
    namespace: openshift-ingress
$ echo "MTcwNCBieXRlcyBsb25n" | base64 -d                                                                                            
1704 bytes long

[guliaka]

Interesting! All other components taking this certificate and working, I can see that console is secured with this certificate. But only single component, authentication, is broken. I looked at cert and it is totally fine. I put cert on two other machines just to validate it - and it is working fine. This cert has full chain in it:

…

-----BEGIN CERTIFICATE----- <-- *.apps.kube-prd.domain.tld ---> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <--- CA intermediary 1 cert---> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <--- CA intermediary 2 cert---> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <--- CA root cert---> -----END CERTIFICATE-----

On Sun, Feb 13, 2022 at 12:11 AM Vadim Rutkovsky ***@***.***> wrote: In namespaces/openshift-ingress-operator/ operator.openshift.io/ingresscontrollers/default.yaml a cert from the secret is used: apiVersion: operator.openshift.io/v1 kind: IngressController metadata: .. name: default namespace: openshift-ingress-operator .. spec: ... defaultCertificate: name: incommon-wc-ingress It looks malformed: $ cat namespaces/openshift-ingress/core/secrets.yaml | grep -C4 "incommon-wc-ingress" tls.key: MTcwNCBieXRlcyBsb25n kind: Secret metadata: creationTimestamp: "2022-02-10T23:46:46Z" name: incommon-wc-ingress namespace: openshift-ingress $ echo "MTcwNCBieXRlcyBsb25n" | base64 -d 1704 bytes long — Reply to this email directly, view it on GitHub <#1107 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBTZBIDEXVVM2TKOIBYCADU25RTPANCNFSM5OGLFCZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>