Failed to fetch clusters. #1746

art-niu · 2023-09-20T15:41:03Z

art-niu
Sep 20, 2023

First of all, thanks for creating the tools to help install OKD.

I am following instructions: https://vrutkovs.eu/posts/okd-disconnected-assisted/ , I got the error, Failed to fetch clusters, when I open the Assisted installer url: http://ops-lab.csd.ca:8080, as I checked the log, and found below error:

2023-09-20T11:15:46.903213793-04:00 2023/09/20 15:15:46 [error] 10#0: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.223.47.225, server: _, request: "GET /api/assisted-install/v2/clusters/ HTTP/1.1", upstream: "http://127.0.0.1:8090/api/assisted-install/v2/clusters/", host: "ops-lab.csd.ca:8080", referrer: "http://ops-lab.csd.ca:8080/clusters"
2023-09-20T11:15:46.903213793-04:00 2023/09/20 15:15:46 [error] 10#0: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.223.47.225, server: _, request: "GET /api/assisted-install/v2/clusters/ HTTP/1.1", upstream: "http://127.0.0.1:8090/api/assisted-install/v2/clusters/", host: "ops-lab.csd.ca:8080", referrer: "http://ops-lab.csd.ca:8080/clusters"

It seems it didn't pick the correct server dns name/ip addr.

Could you please shad some lights on issue?

Thanks,
Arthur

vrutkovs · 2023-09-20T16:46:42Z

vrutkovs
Sep 20, 2023
Maintainer

Check details in okd-configmap.yml:

ASSISTED_SERVICE_HOST: 127.0.0.1:8090
SERVICE_BASE_URL: http://127.0.0.1:8090

In the instruction this is set by sed -i 's;127.0.0.1:8;assisted.vrutkovs.eu:8;g' disconnected-okd-configmap.yml command, but it may be too broad for you

7 replies

art-niu Sep 21, 2023
Author

The firewalld has been disabled. But port 8090 and 8888 are unreachable.

[root@ops-lab ~]# sudo firewall-cmd --list-all
FirewallD is not running
[root@ops-lab ~]# podman ps
CONTAINER ID  IMAGE                                                                    COMMAND               CREATED       STATUS       PORTS                                                                   NAMES
1d0513c8dcde  docker.io/library/registry:2.7.0                                         /etc/docker/regis...  9 days ago    Up 9 days    0.0.0.0:443->443/tcp                                                    ops-lab
00fcf8a9fe08  docker.io/library/python:latest                                          python3 -m http.s...  9 days ago    Up 9 days    0.0.0.0:3000->3000/tcp                                                  image-storage
255d88d5a68c  localhost/podman-pause:4.4.1-1689930982                                                        16 hours ago  Up 16 hours  0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  b7a8397573c2-infra
ee49f5a8a1c4  ops-lab.csd.ca/centos7/postgresql-12-centos7:latest              run-postgresql        16 hours ago  Up 16 hours  0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-db
1cf2dc484eb6  ops-lab.csd.ca/edge-infrastructure/assisted-installer-ui:latest  /deploy/start.sh      16 hours ago  Up 16 hours  0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-ui
[root@ops-lab ~]# telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
^]
telnet> q
Connection closed.
[root@ops-lab ~]# telnet 127.0.0.1 8090
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
[root@ops-lab ~]# telnet 127.0.0.1 8888
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

[root@ops-lab ~]# netstat -tunap|grep LISTEN
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      335513/conmon
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      275934/conmon
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      335513/conmon
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      274654/conmon
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      335513/conmon
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      695/sshd

vrutkovs Sep 21, 2023
Maintainer

Seems assisted-installer-service is not running, just the UI. Check if its in stopped containers - podman ps -a. If yes, check logs

art-niu Sep 21, 2023
Author

You are right. Investigating. Please shad some light on it, it will help me resolve quickly.

[root@ops-lab assisted-service]# podman ps -a
CONTAINER ID  IMAGE                                                                     COMMAND               CREATED       STATUS                   PORTS                                                                   NAMES
1d0513c8dcde  docker.io/library/registry:2.7.0                                          /etc/docker/regis...  10 days ago   Up 10 days               0.0.0.0:443->443/tcp                                                    ops-lab
00fcf8a9fe08  docker.io/library/python:latest                                           python3 -m http.s...  9 days ago    Up 9 days                0.0.0.0:3000->3000/tcp                                                  image-storage
255d88d5a68c  localhost/podman-pause:4.4.1-1689930982                                                         17 hours ago  Up 17 hours              0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  b7a8397573c2-infra
ee49f5a8a1c4  ops-lab.csd.ca/centos7/postgresql-12-centos7:latest               run-postgresql        17 hours ago  Up 17 hours              0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-db
1cf2dc484eb6  ops-lab.csd.ca/edge-infrastructure/assisted-installer-ui:latest   /deploy/start.sh      17 hours ago  Up 17 hours              0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-ui
04da63a3fa91  ops-lab.csd.ca/edge-infrastructure/assisted-image-service:latest  /assisted-image-s...  17 hours ago  Exited (1) 17 hours ago  0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-image-service
2f4539546d55  ops-lab.csd.ca/edge-infrastructure/assisted-service:latest        /assisted-service     17 hours ago  Exited (1) 17 hours ago  0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-service
[root@ops-lab assisted-service]# podman logs 04da63a3fa91
{"file":"/go/src/github.com/openshift/origin/pkg/imagestore/imagestore.go:198","func":"github.com/openshift/assisted-image-service/pkg/imagestore.(*rhcosStore).Populate.func1","level":"info","msg":"Downloading iso from http://ops-lab.csd.ca:3000/fedora-coreos--live.x86_64.iso to /data/rhcos-full-iso-4.13--x86_64.iso","time":"2023-09-20T19:12:02Z"}
{"file":"/go/src/github.com/openshift/origin/pkg/servers/servers.go:91","func":"github.com/openshift/assisted-image-service/pkg/servers.(*ServerInfo).httpListen","level":"info","msg":"Starting http handler on :8888...","time":"2023-09-20T19:12:02Z"}
{"file":"/go/src/github.com/openshift/origin/main.go:80","func":"main.main.func1","level":"fatal","msg":"Failed to populate image store: failed to download http://ops-lab.csd.ca:3000/fedora-coreos--live.x86_64.iso: request to http://ops-lab.csd.ca:3000/fedora-coreos--live.x86_64.iso returned error code 404\n","time":"2023-09-20T19:12:02Z"}
[root@ops-lab assisted-service]# podman logs 2f4539546d55
false
time="2023-09-20T19:12:03Z" level=info msg="Setting log format: text" func=main.InitLogs file="/assisted-service/cmd/main.go:184"
time="2023-09-20T19:12:03Z" level=info msg="Setting Log Level: info" func=main.InitLogs file="/assisted-service/cmd/main.go:190"
time="2023-09-20T19:12:03Z" level=info msg="Starting assisted-service version: 60d8a476e616069062e47e113b581c21daf6c775" func=main.main file="/assisted-service/cmd/main.go:234"
time="2023-09-20T19:12:03Z" level=info msg="Starting bm service" func=main.main file="/assisted-service/cmd/main.go:249"
time="2023-09-20T19:12:03Z" level=fatal msg="Failed to initialize OSImages" func=main.main.func1 file="/assisted-service/cmd/main.go:242" error="Missing value in OSImage for 'version' field (openshift_version: 4.13)"

art-niu Sep 21, 2023
Author

disconnected-okd-configmap.yml disconnected-pod.yml are attached here.
pod.zip

art-niu Sep 21, 2023
Author

FCOS_VERSION and OKD_VERSION were missed in disconnected-okd-configmap.yml

art-niu · 2023-09-21T16:04:50Z

art-niu
Sep 21, 2023
Author

Now, the database is unreachable, the Postgresql container is up but not listening. But didn't find issue with the database:

[root@ops-lab assisted-service]# podman ps -a
CONTAINER ID  IMAGE                                                                     COMMAND               CREATED         STATUS         PORTS                                                                   NAMES
1d0513c8dcde  docker.io/library/registry:2.7.0                                          /etc/docker/regis...  10 days ago     Up 10 days     0.0.0.0:443->443/tcp                                                    ops-lab
00fcf8a9fe08  docker.io/library/python:latest                                           python3 -m http.s...  10 days ago     Up 10 days     0.0.0.0:3000->3000/tcp                                                  image-storage
9d42611f287b  localhost/podman-pause:4.4.1-1689930982                                                         10 seconds ago  Up 10 seconds  0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  facdce35b9e2-infra
f5c4e717ad7f  ops-lab.csd.ca/centos7/postgresql-12-centos7:latest               run-postgresql        10 seconds ago  Up 9 seconds   0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-db
177f7b02e126  ops-lab.csd.ca/edge-infrastructure/assisted-installer-ui:latest   /deploy/start.sh      10 seconds ago  Up 10 seconds  0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-ui
4fb47198bbf9  ops-lab.csd.ca/edge-infrastructure/assisted-image-service:latest  /assisted-image-s...  10 seconds ago  Up 9 seconds   0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-image-service
f7d2683683c9  ops-lab.csd.ca/edge-infrastructure/assisted-service:latest        /assisted-service     10 seconds ago  Up 7 seconds   0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp, 0.0.0.0:8888->8888/tcp  assisted-installer-service
[root@ops-lab assisted-service]# netstat -tunap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      341064/conmon
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      275934/conmon
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      341064/conmon
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      274654/conmon
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      341064/conmon
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      695/sshd
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::22                   :::*                    LISTEN      695/sshd
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/systemd
udp6       0      0 :::111                  :::*                                1/systemd
udp6       0      0 ::1:323                 :::*                                664/chronyd
[root@ops-lab assisted-service]# netstat -tunap |grep 5432

Here is the logs of the container:

[root@ops-lab assisted-service]# podman logs f5c4e717ad7f
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/pgsql/data/userdata -l logfile start

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2023-09-21 15:55:09.617 UTC [36] LOG:  starting PostgreSQL 12.11 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2023-09-21 15:55:09.617 UTC [36] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-09-21 15:55:09.618 UTC [36] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-09-21 15:55:09.629 UTC [36] LOG:  redirecting log output to logging collector process
2023-09-21 15:55:09.629 UTC [36] HINT:  Future log output will appear in directory "log".
 done
server started
/var/run/postgresql:5432 - accepting connections
=> sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
ALTER ROLE
waiting for server to shut down.... done
server stopped
Starting server...
2023-09-21 15:55:10.338 UTC [1] LOG:  starting PostgreSQL 12.11 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2023-09-21 15:55:10.338 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-09-21 15:55:10.338 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-09-21 15:55:10.340 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-09-21 15:55:10.342 UTC [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-09-21 15:55:10.352 UTC [1] LOG:  redirecting log output to logging collector process
2023-09-21 15:55:10.352 UTC [1] HINT:  Future log output will appear in directory "log".

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to fetch clusters. #1746

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Failed to fetch clusters. #1746

art-niu Sep 20, 2023

Replies: 2 comments · 7 replies

vrutkovs Sep 20, 2023 Maintainer

art-niu Sep 21, 2023 Author

vrutkovs Sep 21, 2023 Maintainer

art-niu Sep 21, 2023 Author

art-niu Sep 21, 2023 Author

art-niu Sep 21, 2023 Author

art-niu Sep 21, 2023 Author

art-niu
Sep 20, 2023

Replies: 2 comments 7 replies

vrutkovs
Sep 20, 2023
Maintainer

art-niu Sep 21, 2023
Author

vrutkovs Sep 21, 2023
Maintainer

art-niu Sep 21, 2023
Author

art-niu Sep 21, 2023
Author

art-niu Sep 21, 2023
Author

art-niu
Sep 21, 2023
Author