MLOps - Sharing Hub does not work

[blaziq]

After deployment, the Sharing Hub does not load. It gets stuck with what you can see in the screenshot:

Logs:

$ kubectl logs -n sharinghub sharinghub-5547fcc964-57289
[2024-11-25 14:00:41 +0000] [1] [INFO] Starting gunicorn 23.0.0
[2024-11-25 14:00:41 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2024-11-25 14:00:41 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2024-11-25 14:00:41 +0000] [7] [INFO] Booting worker with pid: 7
[2024-11-25 14:00:41 +0000] [8] [INFO] Booting worker with pid: 8
[2024-11-25 14:00:41 +0000] [9] [INFO] Booting worker with pid: 9
[2024-11-25 14:00:41 +0000] [10] [INFO] Booting worker with pid: 10
[2024-11-25 14:00:41 +0000] [11] [INFO] Booting worker with pid: 11
[2024-11-25 14:00:41 +0000] [12] [INFO] Booting worker with pid: 12
[2024-11-25 14:00:41 +0000] [13] [INFO] Booting worker with pid: 13
[2024-11-25 14:00:41 +0000] [14] [INFO] Booting worker with pid: 14
[2024-11-25 14:00:41 +0000] [15] [INFO] Booting worker with pid: 15
[2024-11-25 14:00:41 +0000] [16] [INFO] Booting worker with pid: 16
[2024-11-25 14:00:41 +0000] [17] [INFO] Booting worker with pid: 17
[2024-11-25 14:00:41 +0000] [18] [INFO] Booting worker with pid: 18
[2024-11-25 14:00:42 +0000] [14] [INFO] Started server process [14]
[2024-11-25 14:00:42 +0000] [14] [INFO] Waiting for application startup.
[2024-11-25 14:00:42 +0000] [14] [INFO] Application startup complete.
[2024-11-25 14:00:43 +0000] [11] [INFO] Started server process [11]
[2024-11-25 14:00:43 +0000] [11] [INFO] Waiting for application startup.
[2024-11-25 14:00:43 +0000] [11] [INFO] Application startup complete.
[2024-11-25 14:00:43 +0000] [9] [INFO] Started server process [9]
[2024-11-25 14:00:43 +0000] [9] [INFO] Waiting for application startup.
[2024-11-25 14:00:43 +0000] [9] [INFO] Application startup complete.
[2024-11-25 14:00:43 +0000] [7] [INFO] Started server process [7]
[2024-11-25 14:00:43 +0000] [7] [INFO] Waiting for application startup.
[2024-11-25 14:00:43 +0000] [7] [INFO] Application startup complete.
[2024-11-25 14:00:43 +0000] [13] [INFO] Started server process [13]
[2024-11-25 14:00:43 +0000] [13] [INFO] Waiting for application startup.
[2024-11-25 14:00:43 +0000] [13] [INFO] Application startup complete.
[2024-11-25 14:00:43 +0000] [16] [INFO] Started server process [16]
[2024-11-25 14:00:43 +0000] [16] [INFO] Waiting for application startup.
[2024-11-25 14:00:43 +0000] [16] [INFO] Application startup complete.
[2024-11-25 14:00:43 +0000] [8] [INFO] Started server process [8]
[2024-11-25 14:00:43 +0000] [8] [INFO] Waiting for application startup.
[2024-11-25 14:00:43 +0000] [8] [INFO] Application startup complete.
[2024-11-25 14:00:43 +0000] [12] [INFO] Started server process [12]
[2024-11-25 14:00:43 +0000] [12] [INFO] Waiting for application startup.
[2024-11-25 14:00:43 +0000] [12] [INFO] Application startup complete.
[2024-11-25 14:00:44 +0000] [10] [INFO] Started server process [10]
[2024-11-25 14:00:44 +0000] [10] [INFO] Waiting for application startup.
[2024-11-25 14:00:44 +0000] [10] [INFO] Application startup complete.
[2024-11-25 14:00:44 +0000] [17] [INFO] Started server process [17]
[2024-11-25 14:00:44 +0000] [17] [INFO] Waiting for application startup.
[2024-11-25 14:00:44 +0000] [17] [INFO] Application startup complete.
[2024-11-25 14:00:44 +0000] [15] [INFO] Started server process [15]
[2024-11-25 14:00:44 +0000] [15] [INFO] Waiting for application startup.
[2024-11-25 14:00:44 +0000] [15] [INFO] Application startup complete.
[2024-11-25 14:00:44 +0000] [18] [INFO] Started server process [18]
[2024-11-25 14:00:44 +0000] [18] [INFO] Waiting for application startup.
[2024-11-25 14:00:44 +0000] [18] [INFO] Application startup complete.

Describe:

$ kubectl describe -n sharinghub  pod sharinghub-5547fcc964-57289
Name:             sharinghub-5547fcc964-57289
Namespace:        sharinghub
Priority:         0
Service Account:  sharinghub
Node:             eoepcak2/10.28.9.16
Start Time:       Mon, 25 Nov 2024 14:00:40 +0000
Labels:           app.kubernetes.io/instance=sharinghub
                  app.kubernetes.io/name=sharinghub
                  pod-template-hash=5547fcc964
Annotations:      cni.projectcalico.org/containerID: e90c666136994318ed93f4c0231adc51c3c0008d060a628ade6609912cade2ce
                  cni.projectcalico.org/podIP: 10.42.127.142/32
                  cni.projectcalico.org/podIPs: 10.42.127.142/32
Status:           Running
IP:               10.42.127.142
IPs:
  IP:           10.42.127.142
Controlled By:  ReplicaSet/sharinghub-5547fcc964
Containers:
  sharinghub:
    Container ID:   containerd://41ff1eda7528099fc7225317afad6da4f8838b2960c42d22edaad558f28b2070
    Image:          eoepca/sharinghub:0.3.0
    Image ID:       docker.io/eoepca/sharinghub@sha256:45bae3b91aa10e31067fee7311f499a44d85c1d6adf5bf99259b2ddcca6c0a0d
    Port:           8000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 25 Nov 2024 14:00:40 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      CONFIG_PATH:                 /home/app/config.yaml
      SESSION_SECRET_KEY:          <set to the key 'session-secret-key' in secret 'sharinghub'>  Optional: false
      GITLAB_OAUTH_CLIENT_ID:      <set to the key 'client-id' in secret 'sharinghub-oidc'>      Optional: true
      GITLAB_OAUTH_CLIENT_SECRET:  <set to the key 'client-secret' in secret 'sharinghub-oidc'>  Optional: true
      GITLAB_OAUTH_DEFAULT_TOKEN:  <set to the key 'default-token' in secret 'sharinghub-oidc'>  Optional: true
      S3_ACCESS_KEY:               <set to the key 'access-key' in secret 'sharinghub-s3'>       Optional: true
      S3_SECRET_KEY:               <set to the key 'secret-key' in secret 'sharinghub-s3'>       Optional: true
    Mounts:
      /home/app/config.yaml from sharinghub-config (rw,path="config.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ntxrm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  sharinghub-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sharinghub-config
    Optional:  false
  kube-api-access-ntxrm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age    From               Message
  ----     ------     ----   ----               -------
  Normal   Scheduled  8m18s  default-scheduler  Successfully assigned sharinghub/sharinghub-5547fcc964-57289 to eoepcak2
  Normal   Pulled     8m18s  kubelet            Container image "eoepca/sharinghub:0.3.0" already present on machine
  Normal   Created    8m18s  kubelet            Created container sharinghub
  Normal   Started    8m18s  kubelet            Started container sharinghub
  Warning  Unhealthy  8m16s  kubelet            Readiness probe failed: Get "http://10.42.127.142:8000/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)