Connot pull from OKD cluster via pod or oc command with error x509: certificate is valid for ***Clusterdomain *** , not *** registry-1.docker.io ***

[glennodickson]

Hi

I have a basic installation cluster OKD4.13. Nothing is installed on it just yet.

Describe the bug
It seems that when pulling from repositories such as docker or quay.io it errors saying the unsigned certificate is incompatible with the docker/quay certificate. Also tried deploying springboot application using JKube and received the same error message.

Unsure why pulling it is concerned with the cluster's unsigned certificate when it is accessing and handshaking with the repos certificate which is signed.

The below errors show when pulling from the docker (or quay) repos:
(x509: certificate is valid for *.apps.test.fritz.box, not registry-1.docker.io)

Version
4.13.0-0.okd-2023-08-04-164726
UPI

How reproducible
Execute below commands for Docker and Quay.

Accessing Docker

Command:
oc import-image myproject/myimage-ref-source:mytag --from="docker.io/balazsszeti/hello:sleeper" --confirm

Output:

error: tag sleeper failed: Internal error occurred: docker.io/balazsszeti/hello:sleeper: Get "https://registry-1.docker.io/v2/": x509: certificate is valid for *.apps.test.fritz.box, not registry-1.docker.io
imagestream.image.openshift.io/myimage-ref-source imported with errors

Name:                   myimage-ref-source
Namespace:              test
Created:                7 minutes ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2023-08-10T15:00:56Z
Image Repository:       default-route-openshift-image-registry.apps.test.fritz.box/test/myimage-ref-source
Image Lookup:           local=false
Unique Images:          0
Tags:                   1

mytag
  tagged from docker.io/balazsszeti/hello:sleeper

  ! error: Import failed (InternalError): Internal error occurred: docker.io/balazsszeti/hello:sleeper: Get "https://registry-1.docker.io/v2/": x509: certificate is valid for *.apps.test.fritz.box, not registry-1.docker.io

Accessing Quay.io

Command:

oc import-image quay.io/andreipope/podman-nuxtjs-demo --confirm

Output:

       
error: tag latest failed: Internal error occurred: quay.io/andreipope/podman-nuxtjs-demo:latest: Get "https://quay.io/v2/": x509: certificate is valid for *.apps.test.fritz.box, not quay.io
imagestream.image.openshift.io/podman-nuxtjs-demo imported with errors

Name:                   podman-nuxtjs-demo
Namespace:              test
Created:                1 second ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2023-08-10T15:59:14Z
Image Repository:       default-route-openshift-image-registry.apps.test.fritz.box/test/podman-nuxtjs-demo
Image Lookup:           local=false
Unique Images:          0
Tags:                   1

latest
  tagged from quay.io/andreipope/podman-nuxtjs-demo

  ! error: Import failed (InternalError): Internal error occurred: quay.io/andreipope/podman-nuxtjs-demo:latest: Get "https://quay.io/v2/": x509: certificate is valid for *.apps.test.fritz.box, not quay.io
      1 second ago


error: imported completed with errors

View Certificate

Command:

openssl s_client -connect quay.io:443

Output:

CONNECTED(00000003)
depth=1 CN = ingress-operator@1690637905
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 CN = ingress-operator@1690637905
verify return:1
depth=0 CN = *.apps.test.fritz.box
verify return:1
---
Certificate chain
 0 s:CN = *.apps.test.fritz.box
   i:CN = ingress-operator@1690637905
 1 s:CN = ingress-operator@1690637905
   i:CN = ingress-operator@1690637905
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDYTCCAkmgAwIBAgIIXhDDkfnqsyIwDQYJKoZIhvcNAQELBQAwJjEkMCIGA1UE
AwwbaW5ncmVzcy1vcGVyYXRvckAxNjkwNjM3OTA1MB4XDTIzMDcyOTEzMzgyNVoX
DTI1MDcyODEzMzgyNlowIDEeMBwGA1UEAwwVKi5hcHBzLnRlc3QuZnJpdHouYm94
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtBOq4/SKvLvI625mkaQQ
diG8tr1lHpb52YttdsufzBWli4ceYdpnGxifOBwlvB6mIpxhJoyBHahh+IaOi+IN
G39UU0o0QtTxPRm38NDPimv90CZ/AtDX5I5ewCc5JvgU6mokw3Y3lSDSo8GBRVdN
nX+uFl34yGhnIPkgZOPXm9EgFLydH/29kSFBySrZotK4GpbaOx85HPXtlA1CpbfB
++1FVOEkOt/cMeX6VzzyH9Y6YMhKoj8B3uzPQ6W73wKRLg/dDDZPct+woahADpB7
hcH7MRwTQay0OH4r5WVEZ5wnlvas88YbMdxi1rh41zJ5dOS0UhkzcV2oGKex4CL5
RQIDAQABo4GYMIGVMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcD
ATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBRVz2W7XseHlU/swAwB4eY/BCnLSjAf
BgNVHSMEGDAWgBR5yE2CUFagMZI0YuSbIlHIfwQbcDAgBgNVHREEGTAXghUqLmFw
cHMudGVzdC5mcml0ei5ib3gwDQYJKoZIhvcNAQELBQADggEBAOrPwMrkTESB+ouv
Wd3TCpT3Z73k6u5izP5dO7mVM9o7DbwTdelZvsYa0Z7DX8NsFVSbfDUjqpmTCRKs
2J4Y+LYLvDfIhqUbKFBSswQT/IZKEGvG9Jwy1Hm8jOmu4PAcGVu0KZ5iyUWSymQV
R8sYxxcBUzdMR8ZJyWh3xOv6bexoM7mS1Ca88N1lZhlrNY3RmlFCD1rG23VLgvEw
gHXRko4ulyk5AM6T98XOLKVH2sa0pXjoR637Bc+SXj/ADqw7WmBSnk1HtgiUG1BR
kWNrb7YArzz7Sf3QRwA3mo2jGCxVlQeOV1uZawPTxymgVKx3mg3Sy/az001XTA3t
EMmTs1U=
-----END CERTIFICATE-----
subject=CN = *.apps.test.fritz.box

issuer=CN = ingress-operator@1690637905

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2202 bytes and written 369 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 99057A87CADBB21A71E211BD8C9726270706A4E7A684BBC7AB30CD942F075F72
    Session-ID-ctx: 
    Resumption PSK: 6B82072ECDA3FFA52945798DC54586779160154F4E290C93B63C775F481D2186
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 36 76 1b d7 bc 25 b9 46-7f 55 be b3 b5 5c f9 d5   6v...%.F.U...\..
    0010 - ae 21 f1 ff cd 3d 4d 7a-a4 96 5d 7c 16 fb 81 f6   .!...=Mz..]|....
    0020 - 5f 96 cc f0 60 63 73 ff-8e e9 de 03 e1 0f b8 79   _...`cs........y
    0030 - 04 64 bd 97 df 2a af 35-8d d4 2d 8d 1a 4d 4e 47   .d...*.5..-..MNG
    0040 - fa 27 e6 89 91 1f 54 28-b6 b1 15 5d d2 14 5d 30   .'....T(...]..]0
    0050 - 45 4b de 3b 7a 36 2d a1-b5 6e 20 da f8 18 79 9d   EK.;z6-..n ...y.
    0060 - 34 a3 1a 81 95 b9 2c 09-dd 30 dd 17 a7 77 48 69   4.....,..0...wHi
    0070 - 83 6c af 47 86 13 31 a8-06 c7 49 80 46 2d 32 22   .l.G..1...I.F-2"
    0080 - 5c d3 e5 0f a9 d5 d4 dd-ff 6b 93 38 f5 b4 a0 97   \........k.8....
    0090 - 11 38 46 54 9f b1 77 c2-58 5a d1 ed ee 2b d0 70   .8FT..w.XZ...+.p
    00a0 - aa 40 8e 25 41 5f 61 5d-30 3c 2c c0 16 be 52 7e   .@.%A_a]0<,...R~
    00b0 - 00 f4 e0 9b f3 29 1c ef-66 70 5c 31 4f bb 97 14   .....)..fp\1O...

    Start Time: 1691564004
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: FF6CB57D902C1391977AD95BE47CA448105271D7AAAAE6DB083C0CDFDEE33210
    Session-ID-ctx: 
    Resumption PSK: 128A76302E9C5F01726BA97D005D900F786D59C1157161B5D884E083BE2F5D1A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 36 76 1b d7 bc 25 b9 46-7f 55 be b3 b5 5c f9 d5   6v...%.F.U...\..
    0010 - e8 51 12 d0 44 c7 a0 f9-96 1e 3e 15 23 b3 ce 99   .Q..D.....>.#...
    0020 - b6 5d a8 d1 3d 2c 29 c1-9c 99 c0 de 10 41 30 18   .]..=,)......A0.
    0030 - 92 23 09 23 30 b1 d3 73-ac f3 14 6c 4c 16 13 35   .#.#0..s...lL..5
    0040 - 3e 70 93 09 46 4d 9a 54-af fc 59 30 c4 55 f6 97   >p..FM.T..Y0.U..
    0050 - f3 b8 69 47 37 5d 8e b0-a0 dc f9 92 d7 5e ed 14   ..iG7].......^..
    0060 - a7 35 6a 92 a8 1e cc 58-bf 54 57 2b b9 cd 25 e0   .5j....X.TW+..%.
    0070 - 17 d4 9b b5 85 0a 6a 4f-2a 61 e5 e0 49 a5 b9 0e   ......jO*a..I...
    0080 - d6 55 bc 06 e9 e3 f9 65-46 90 22 17 13 72 ce e1   .U.....eF."..r..
    0090 - 75 12 74 13 c3 b1 21 55-73 94 cf 4b 49 8b 62 e9   u.t...!Us..KI.b.
    00a0 - e4 10 d7 26 13 c8 80 87-87 90 7c bb bb 5e 0c df   ...&......|..^..
    00b0 - 1e 44 ff f4 4d 1d 08 50-d0 14 47 2e 26 2d 3b 88   .D..M..P..G.&-;.

    Start Time: 1691564004
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
HTTP/1.1 408 Request Time-out
content-length: 110
cache-control: no-cache
content-type: text/html
connection: close

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed

TIA
Glenn

[vrutkovs]

x509: certificate is valid for *.apps.test.fritz.box, not registry-1.docker.io
x509: certificate is valid for *.apps.test.fritz.box, not quay.io

Perhaps docker.io and quay.io is not resolving correctly on your cluster? Check that dig quay.io docker.io and dig quay.io docker.io @1.1.1.1 are similar on the node

[rvanderp3]

I think this may have something to do with search domains for lookups within the pod. I've run in to this before. if you run nslookup quay.io you might get a different result vs dig. I believe the issue is that when looking up quay.io or docker.io, quay.io.test.fritz.box might be getting looked up. @glennodickson , by chance, do you have a wildcard record for *.test.fritz.box?

[glennodickson]

Hi
Yes you are right both the dig and nslookup return different results with quay.io.test.fritz.box and doicker.io.test.fritz.box getting looked up. However, I don't think I have a wildcard in the below dns records.

Here are my dns records definitions:

address=/test.fritz.box/192.168.179.73
address=/jumpbox.test.fritz.box/192.168.179.70
address=/lb.test.fritz.box/192.168.179.73
address=/dns.test.fritz.box/192.168.179.74
address=/bootstrap.test.fritz.box/192.168.179.72
address=/master0.test.fritz.box/192.168.179.80
address=/master1.test.fritz.box/192.168.179.81
address=/master2.test.fritz.box/192.168.179.82
address=/infra0.test.fritz.box/192.168.179.83
address=/infra1.test.fritz.box/192.168.179.84
address=/infra2.test.fritz.box/192.168.179.85
address=/app0.test.fritz.box/192.168.179.86
address=/app1.test.fritz.box/192.168.179.87
address=/app2.test.fritz.box/192.168.179.88
address=/store0.test.fritz.box/192.168.179.89
address=/store1.test.fritz.box/192.168.179.90
address=/store2.test.fritz.box/192.168.179.91
Here is the output:

Docker

dig docker

; <<>> DiG 9.11.36-RedHat-9.11.36-3.el8_6.1 <<>> docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27226
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; COOKIE: 87b6ab1642efa03b (echoed)
;; QUESTION SECTION:
;docker.io.                     IN      A

;; ANSWER SECTION:
docker.io.              28      IN      A       18.206.20.10
docker.io.              28      IN      A       3.228.146.75
docker.io.              28      IN      A       18.210.197.188

;; Query time: 0 msec
;; SERVER: 172.30.0.10#53(172.30.0.10)
;; WHEN: Fri Aug 11 12:03:26 UTC 2023
;; MSG SIZE  rcvd: 125

nslookup docker.io

Server:         172.30.0.10
Address:        172.30.0.10#53

Name:   docker.io.test.fritz.box
Address: 192.168.179.73

Quay.io

dig quay.io

; <<>> DiG 9.11.36-RedHat-9.11.36-3.el8_6.1 <<>> quay.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9684
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; COOKIE: f32a4bde021ccd09 (echoed)
;; QUESTION SECTION:
;quay.io.                       IN      A

;; ANSWER SECTION:
quay.io.                60      IN      A       44.206.35.102
quay.io.                60      IN      A       35.174.158.243
quay.io.                60      IN      A       44.197.11.57
quay.io.                60      IN      A       3.213.170.213
quay.io.                60      IN      A       54.197.124.39
quay.io.                60      IN      A       34.192.130.188
quay.io.                60      IN      A       54.198.90.191
quay.io.                60      IN      A       107.22.98.229

;; Query time: 66 msec
;; SERVER: 172.30.0.10#53(172.30.0.10)
;; WHEN: Fri Aug 11 11:45:51 UTC 2023
;; MSG SIZE  rcvd: 232

nslookup quay.io

Server:         172.30.0.10
Address:        172.30.0.10#53

Name:   quay.io.test.fritz.box
Address: 192.168.179.73

[rvanderp3]

Hmmm, well something is resolving *.test.fritz.box or we would be seeing an nxdomain for quay.io.test.fritz.box. On your DNS host, you might use tcpdump to trace DNS lookups to see where that might be getting resolved. Something like:

tcpdump -n udp and port 53

[glennodickson]

Hi Richard
Thanks for your suggestions. I've tested the DNS host just not sure if it is correct. Also below. I've reviewed the process flow the pods uses to resolve a query. Any further suggestions would be very appreciated.

Output from Testing DNS Host (192.168.179.74)

When testing the dnsmasq server using the suggested command I noted that the output contains the URL openshift.apps.test.fritz.box.test.fritz.box which has test.fritz.box appended. This is the FQDN mentioned in OCP DNS Cluster Pod (see last section). I was expecting that openshift.apps.test.fritz.box.

$ tcpdump -n udp and port 53

dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
00:22:38.782458 IP 192.168.179.80.53280 > 192.168.179.74.domain: 26016+ [1au] A? oauth-openshift.apps.test.fritz.box.test.fritz.box. (79)
00:22:38.782633 IP 192.168.179.74.domain > 192.168.179.80.53280: 26016* 1/0/0 A 192.168.179.73 (84)
00:22:38.782973 IP 192.168.179.80.33495 > 192.168.179.74.domain: 50246+ [1au] AAAA? oauth-openshift.apps.test.fritz.box.test.fritz.box. (79)
00:22:38.783062 IP 192.168.179.74.domain > 192.168.179.80.33495: 50246 0/0/0 (68)
00:22:49.038851 IP 192.168.179.82.50862 > 192.168.179.74.domain: 51376+ AAAA? console-openshift-console.apps.test.fritz.box.test.fritz.box. (78)
00:22:49.038917 IP 192.168.179.82.34739 > 192.168.179.74.domain: 4019+ A? console-openshift-console.apps.test.fritz.box.test.fritz.box. (78)
00:22:49.038994 IP 192.168.179.74.domain > 192.168.179.82.50862: 51376 0/0/0 (78)

Pod don't (and won't let it be installed) have tcpdump and therefore, could not trial this.

DNS Resolution Flow

DNS resolution it seems that pod follows the below flow:

    pod   ----------->  OCP DNS Cluster Service  -----------> OCP DNS Cluster Pod   --------->  Dnsmsq Server
                                  (172.30.0.10)                                                                                             (192.168.179.74)

When testing with DIG it works in the pod, OCP DNS Cluster Pod and Dnsmsq Server.
Using nsliookup from pod fails however OCP DNS Cluster Pod and Dnsmsq Server successfully return the correct IP for quay and docker.
I would have thought the "pod" would successfully return the correct value.

Details of DNS Cluster service:
openshift-dns dns-default ClusterIP 172.30.0.10 <none> 53/UDP,53/TCP,9154/TCP 14d

List of OCP Cluster DNS pods:

openshift-dns        dns-default-2585t    2/2     Running     12              14d     10.128.0.40      master0   <none>           <none>
openshift-dns        dns-default-c5xg8    2/2     Running     12              14d     10.128.4.7       infra0    <none>           <none>
openshift-dns        dns-default-jmflr    2/2     Running     12              14d     10.128.3.14      master1   <none>           <none>
openshift-dns        dns-default-xntt2    2/2     Running     12              14d     10.128.1.11      master2   <none>           <none>
openshift-dns        dns-default-z74mq    2/2     Running     12              14d     10.128.2.11      infra2    <none>           <none>
openshift-dns        dns-default-zz7p2    2/2     Running     12              14d     10.128.5.3       infra1    <none>           <none>

Resolv.confg
Pod: Resolv.config

search openshift-image-registry.svc.cluster.local svc.cluster.local cluster.local test.fritz.box
nameserver 172.30.0.10
options ndots:5

OCP DNS Cluster Pod: resolv.conf file

search test.fritz.box
nameserver 192.168.179.74

TiA
Glenn

[rvanderp3]

Hi Glenn -
The DNS resolver pods will add the cluster base domain as a search domain. I believe since the initial domain name(has 4 dots) is less than ndots, it appends the search domain. I think in this case, in order for this to work, 192.168.179.74 needs to return NXDOMAIN rather than an A record for oauth-openshift.apps.test.fritz.box.test.fritz.box.

00:22:38.782458 IP 192.168.179.80.53280 > 192.168.179.74.domain: 26016+ [1au] A? oauth-openshift.apps.test.fritz.box.test.fritz.box. (79)
00:22:38.782633 IP 192.168.179.74.domain > 192.168.179.80.53280: 26016* 1/0/0 A 192.168.179.73 (84)

[glennodickson]

Hi

Thanks heaps. I found the issue was that I had this line in the dnsmasq.conf file:
address=/test.fritz.box/192.168.179.73

Taking this out solved the issue.

Thanks so much.

Regards
Glenn