Unable to enable DNS for nebari

[shauryagoel]

We deployed nebari on GCP, also assigned the traefik ingress public IP to a domain on GCP. However, when accessing the nebari services like conda-store or keycloak via the domain, I am getting a 404 not found error. But, the same works when using the public IP.
On inspection of the traefik logs, I can see the following-

time="2025-02-06T09:04:09Z" level=error msg="Secret dev/ does not exist" providerName=kubernetescrd TLSStore=default namespace=dev
time="2025-02-06T09:04:09Z" level=error msg="subset not found for dev/nebari-dask-gateway-gateway-api" ingress=dask-gateway namespace=dev providerName=kubernetescrd
time="2025-02-06T09:04:09Z" level=error msg="subset not found for dev/nebari-grafana" providerName=kubernetescrd ingress=grafana-ingress-route namespace=dev
time="2025-02-06T09:04:09Z" level=error msg="subset not found for dev/nebari-conda-store-minio" ingress=minio-api providerName=kubernetescrd namespace=dev
time="2025-02-06T09:04:09Z" level=error msg="Cannot create service: subset not found" namespace=dev serviceName=nebari-jupyterhub-sftp servicePort="{0 8023 }" providerName=kubernetescrd ingress=jupyterhub-sftp-ingress
time="2025-02-06T09:04:09Z" level=error msg="the service \"dev-jupyterhub-sftp-ingress-673acf455cb2dab0b43a@kubernetescrd\" does not exist" entryPointName=sftp routerName=dev-jupyterhub-sftp-ingress-673acf455cb2dab0b43a@kubernetescrd
time="2025-02-06T09:04:09Z" level=error msg="the router dev-argo-workflows-2d3c9c221bdce013a42e@kubernetescrd uses a non-existent resolver: default"
time="2025-02-06T09:04:09Z" level=error msg="the router dev-argo-workflows-fdf5679802513aa8a816@kubernetescrd uses a non-existent resolver: default"
time="2025-02-06T09:04:09Z" level=error msg="the router dev-keycloak-http-3724f02afbf6291a46b5@kubernetescrd uses a non-existent resolver: default"
time="2025-02-06T09:04:09Z" level=error msg="the router dev-argo-workflows-adcf40eecbee5611bad6@kubernetescrd uses a non-existent resolver: default"
time="2025-02-06T09:04:09Z" level=error msg="the router dev-conda-store-server-764039fc9f27b70c3daf@kubernetescrd uses a non-existent resolver: default"
time="2025-02-06T09:04:09Z" level=error msg="the router dev-jupyterhub-764eecf1fdedff142224@kubernetescrd uses a non-existent resolver: default"
time="2025-02-06T09:04:09Z" level=error msg="the router dev-forwardauth-846beb2aed2923056972@kubernetescrd uses a non-existent resolver: default"
time="2025-02-06T09:04:09Z" level=error msg="the router dev-jupyterhub-e0677d2746cda05deab2@kubernetescrd uses a non-existent resolver: default"
time="2025-02-06T09:04:11Z" level=error msg="Secret dev/ does not exist" providerName=kubernetescrd TLSStore=default namespace=dev
time="2025-02-06T09:04:11Z" level=error msg="subset not found for dev/nebari-dask-gateway-gateway-api" providerName=kubernetescrd ingress=dask-gateway namespace=dev
time="2025-02-06T09:04:11Z" level=error msg="subset not found for dev/nebari-grafana" providerName=kubernetescrd ingress=grafana-ingress-route namespace=dev
time="2025-02-06T09:04:11Z" level=error msg="subset not found for dev/nebari-conda-store-minio" namespace=dev providerName=kubernetescrd ingress=minio-api
time="2025-02-06T09:04:11Z" level=error msg="Cannot create service: subset not found" namespace=dev serviceName=nebari-jupyterhub-sftp providerName=kubernetescrd servicePort="{0 8023 }" ingress=jupyterhub-sftp-ingress

Can someone help me in debugging this.

[marcelovilla]

Hey @shauryagoel, can you share a sanitized version of your nebari-config.yaml file?

[shauryagoel]

This is the short version of the nebari config-

---
provider: gcp
namespace: dev
nebari_version: 2024.12.1
project_name: nebari-new
ci_cd:
  type: none
terraform_state:
  type: remote
security:
  keycloak:
    initial_root_password: <INITIAL_PASSWORD>
  authentication:
    type: password
theme:
  jupyterhub:
    hub_title: Nebari - nebari-new
    welcome: Welcome! Learn about Nebari's features and configurations in <a href="https://www.nebari.dev/docs/welcome">the
      documentation</a>. If you have any questions or feedback, reach the team on
      <a href="https://www.nebari.dev/docs/community#getting-support">Nebari's support
      forums</a>.
    hub_subtitle: Your open source data science platform, hosted on Google Cloud Platform
google_cloud_platform:
  kubernetes_version: 1.29.12-gke.1120001
  region: asia-south1
  node_groups:
    general:
      instance: e2-standard-8
      min_nodes: 1
      max_nodes: 1
      preemptible: false
      labels: {}
      guest_accelerators: []
    user:
      instance: e2-standard-4
      min_nodes: 0
      max_nodes: 5
      preemptible: true
      labels: {}
      guest_accelerators: []
    user-gpu-t4:
      instance: n1-standard-4
      min_nodes: 0
      max_nodes: 5
      preemptible: true
      labels: {}
      guest_accelerators:
        - name: nvidia-tesla-t4
          count: 1
    worker:
      instance: e2-standard-4
      min_nodes: 0
      max_nodes: 5
      preemptible: false
      labels: {}
      guest_accelerators: []
  project: <GCP_PROJECT>
jhub_apps:
  enabled: true
jupyterlab:
  idle_culler:
    server_shutdown_no_activity_timeout: 60
    kernel_cull_idle_timeout: 30
    terminal_cull_inactive_timeout: 30
dns:
  provider: none
profiles:
  jupyterlab:
    - display_name: T4 GPU Instance
      description: 3 cpu / 11GB RAM / 1 Nvidia T4 GPU
      kubespawner_override:
        image: quay.io/nebari/nebari-jupyterlab-gpu:latest
        cpu_guarantee: 3
        cpu_limit: 3
        mem_guarantee: 11G
        mem_limit: 11G
        node_selector:
          cloud.google.com/gke-nodepool: user-gpu-t4
        extra_resource_limits:
          nvidia.com/gpu: 1
    - display_name: Small Instance
      description: Stable environment with 2 cpu / 8 GB ram
      kubespawner_override:
        cpu_guarantee: 1.5
        cpu_limit: 2
        mem_guarantee: 5G
        mem_limit: 8G
        node_selector:
          cloud.google.com/gke-nodepool: user
    - display_name: Medium Instance
      description: Stable environment with 4 cpu / 16 GB ram
      kubespawner_override:
        cpu_guarantee: 4
        cpu_limit: 3
        mem_guarantee: 16G
        mem_limit: 10G
        node_selector:
          cloud.google.com/gke-nodepool: user

[marcelovilla]

Thanks. I don't see you having a domain in the config. Can you try adding that? Also, what DNS provider are you using?

[shauryagoel]

I am using google cloud DNS.
I am following this guide.
Where should I put the domain in the config? Can you specify the docs?

[marcelovilla]

Just add an entry like this in the config file (using your real domain, of course):

provider: gcp
namespace: dev
nebari_version: 2024.12.1
project_name: nebari-new
domain: your.domain.io

The Deploy Nebari on GCP guide mentions the domain but there's not really much more to it. Given that you already setup Google Cloud DNS, you can simply add the Load Balancer IP to a new record (A record in this case) on your DNS provider. You can re-deploy Nebari after modifying the config and you should get a message like the one in the guide you're following:

Take IP Address 12.312.312.312 and update DNS to point to "your.domain.io" [Press Enter when Complete]

Let me know if this works.

[shauryagoel]

Thanks for the help. That is working.

[marcelovilla]

Glad it worked.

As as side note, do you feel the docs were not particularly helpful in this case? Do you think they could be more clear? Happy to get some feedback on how to improve them so it's easier for others trying to setup Nebari with different DNS providers.

[shauryagoel]

I think that typing "domain" in the search bar on the Nebari website, we should get some results specific to the domain config option in nebari-config.yaml.
Also, I think there should be a page where all the nebari-config.yaml flags are mentioned (or maybe I was not able to find it).
If that were the case, then, this discussion could have been avoided.

[marcelovilla]

Thanks for the feedback @shauryagoel! I definitely agree the search bar does not yield the best results. We need to add a page that lists all of the available config options of the nebari-config.yaml file, and we're also planning to add several examples of ready-to-use configs for different providers and different scenarios (e.g. deployment with GPUs on AWS or a low cost deployment on GCP).

In the meantime, you can also generate a file with all flags by using nebari init with the --explicit flag.