add service account with allow-app-sharing-role permissions

[Adam-D-Lewis]

Reference Issues or PRs

In order for startup apps to be used by nebari, we need to create a service account with appropriate permissions to create the apps and share them with others. One issue this caused was the auth state for the service account doesn't get populated and it is needed in our custom Spawner code. This PR updates the service account code during the pre-spawn-hook.

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

How to test this PR?

1. Do a nebari deployment with the following config defined in the nebari config file.

jhub_apps:
  enabled: true
  overrides: {
    startup_apps: [
      {
        "username": "service-account-jupyterhub",  # app will be created by this user
        "servername": "my-startup-server",  # specify a unique server name
        "user_options": {
            "display_name": "My Startup Server",
            "description": "description",
            "thumbnail": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mNkYPhfz0AEYBxVSF+FAP5FDvcfRYWgAAAAAElFTkSuQmCC",  # base64 encoded image data to use for thumbnail
            "filepath": "panel_basic.py",  # local file or path within git repo
            "framework": "panel",
            "public": False,  # Whether or not app is publicly accessible without authentication
            "keep_alive": False,  # Whether or not to shut down app after a period of idleness
            "env": {"MY_ENV_VAR": "MY_VALUE"},
            "repository": {"url": "https://github.com/nebari-dev/jhub-apps-from-git-repo-example.git"},  # specify if pulling app from git repo
            "conda_env": "global-panelenv",
            "profile": "small-instance",
            "share_with": {
              "users": [], 
              "groups": ["/admin"]
            },
        },
      },
    ]
  }

2. Then create this conda env after deployment in the global namespace

name: panelenv
channels:
  - conda-forge
dependencies:
  - panel
  - jhsingle-native-proxy
  - bokeh-root-cmd
  - ipykernel
variables: {}

3. Then create an admin user and give the user the jupyterhub client's "allow-app-sharing-role" role. Log in and make sure you can see my-startup-server listed as a shared app. Then open it and ensure it opens correctly.

Any other comments?

[Adam-D-Lewis]

I'm seeing some issues right now. The issue is the "jhub-apps-sa" user which creates the startup_apps needs to have logged in before the app server can be started successfully. In the current design, the "jhub-apps-sa" user is meant to act more like a service account and not log in interactively as users do. The issues I've seen so far are in the code that is run by the Spawner to set preferred username and render profiles, but it's possible there are more. In those instances, the auth state for the "jhub-apps-sa" user is None before initial log in so an error is thrown once we try to access info in the auth_state object in those methods.

Some possible ideas on how to fix this:

• if the auth_state is None, then go ask keycloak for the needed info? (login on behalf of the user maybe or some other way?)
• log the "jhub-apps-sa" user in somehow on startup app creation?
• I could add startup_app: True to the user options dict used by jhub_apps. I could then modify our problematic spawner code which requires auth_state to only run if the server starting up isn't a startup app.

@krassowski any thoughts on how best to do this?

[krassowski]

This might be too radical and not the kind of suggestion you are asking for, but could it be solved on the jhub-apps level? I mean jhub-apps is meant to be auth-provider agnostic so it should be possible to make this work without touching keycloak at all? This might be just my PTSD from figuring out keycloak piping last time speaking. I recall @aktech also had pleasure to work on keycloak integration - he may have better ideas.

[krassowski]

Some possible ideas on how to fix this:

• if the auth_state is None, then go ask keycloak for the needed info? (login on behalf of the user maybe or some other way?)
• log the "jhub-apps-sa" user in somehow on startup app creation?
• I could add startup_app: True to the user options dict used by jhub_apps. I could then modify our problematic spawner code which requires auth_state to only run if the server starting up isn't a startup app.

If staying with this approach I would probably try the solutions in that exact order. I am not sure if the last one will work if you need to do anything beyond the configuration being set (i.e. whether server will actually spawn if you do not have auth_state).

[aktech]

This might be too radical and not the kind of suggestion you are asking for, but could it be solved on the jhub-apps level? I mean jhub-apps is meant to be auth-provider agnostic so it should be possible to make this work without touching keycloak at all?

That would be ideal, indeed. The way we are using the spawner here, expects the user to be logged in (hence populating auth_state) once before creating a server, which makes sense from jupyterhub pov as the servers are created by humans instead of robots.

This (startup apps) works in jhub-apps (without nebari) by default unless the spawner needs auth_state (which is the case here). Since its a feature of jhub-apps to provide the ability to create init apps on startup regardless of Authenticator or spawner, this should be handled in jhub-apps if possible (this is a big if though), you might need to pass in some kind of keycloak auth details in the JApps config, that might make it possible.

We can try to see if we can somehow call the authenticate method of the Authenticator in jhub-apps for the user, to populate the auth state before starting startup apps, but yes this might not be possible at all, in that case what you suggested in this comment #2917 (comment), sounds like a good approach, and I agree with @krassowski the last one won't work, as you need groups info to create nfs mounts.

Also, another thing to note here, is supporting this in jhub-apps might be tricky (if its possible), its probably ok to just go for your approach and we can tackle it in jhub-apps later, if time is of essence here.

[Adam-D-Lewis]

I think the ideal solution is to create the startup apps up using jupyterhub-service-account which is service account associated with the jupyterhub Keycloak client. We already do the authenticatation needed in Nebari's KeycloakOAuthenticator. We just need to set auth_state for that service account after authentication. I'm not clear on how to set auth state for jupyterhub-service-account, but I'll dig in to the jupyterhub and jupyterhub/OAuthentication code further.

import json
import urllib.parse
import requests
import jwt

# def get_token():
client_id = "jupyterhub"
client_secret = "<my-secret>"
token_url = "https://github-actions.nebari.dev/auth/realms/nebari/protocol/openid-connect/token"

body = urllib.parse.urlencode({
    "client_id": client_id,
    "client_secret": client_secret,
    "grant_type": "client_credentials",
})

headers = {
    'Content-Type': 'application/x-www-form-urlencoded'
}

response = requests.post(token_url, data=body, headers=headers, verify=False)
data = response.json()

# Get the token
token = data["access_token"]

# Decode and print token contents
decoded = jwt.decode(token, options={"verify_signature": False})
print(json.dumps(decoded, indent=2))

yields

{
  "exp": 1737998505,
  "iat": 1737998205,
  "jti": "9827b1e3-9612-4341-8076-22e99d2ccb04",
  "iss": "https://github-actions.nebari.dev/auth/realms/nebari",
  "aud": [
    "realm-management",
    "grafana",
    "argo-server-sso",
    "conda_store",
    "account"
  ],
  "sub": "c1da7cbd-3150-42ae-8b19-c9b4304f2054",
  "typ": "Bearer",
  "azp": "jupyterhub",
  "acr": "1",
  "realm_access": {
    "roles": [
      "offline_access",
      "default-roles-nebari",
      "uma_authorization"
    ]
  },
  "resource_access": {
    "realm-management": {
      "roles": [
        "view-realm",
        "view-users",
        "view-clients",
        "query-clients",
        "query-groups",
        "query-users"
      ]
    },
    "jupyterhub": {
      "roles": [
        "allow-read-access-to-services-role",
        "jupyterhub_developer",
        "allow-group-directory-creation-role"
      ]
    },
    "grafana": {
      "roles": [
        "grafana_viewer"
      ]
    },
    "argo-server-sso": {
      "roles": [
        "argo-viewer"
      ]
    },
    "conda_store": {
      "roles": [
        "conda_store_developer"
      ]
    },
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "email profile",
  "clientHost": "10.244.0.1",
  "email_verified": false,
  "clientId": "jupyterhub",
  "roles": [
    "view-realm",
    "view-users",
    "view-clients",
    "query-clients",
    "query-groups",
    "query-users",
    "allow-read-access-to-services-role",
    "jupyterhub_developer",
    "allow-group-directory-creation-role",
    "grafana_viewer",
    "argo-viewer",
    "conda_store_developer",
    "manage-account",
    "manage-account-links",
    "view-profile"
  ],
  "groups": [
    "/analyst",
    "/users"
  ],
  "preferred_username": "service-account-jupyterhub",
  "clientAddress": "10.244.0.1"
}

which has the group membership for the service account, preferred_username, and permissions similar to a normal user.

[Adam-D-Lewis]

I haven't been able to successfully set auth state for the service account, (Update: resolved by commit 110b0ee (next commit)) but just looking up the service account's info during the spawner code in this commit seems to work.

[Adam-D-Lewis]

Before we only allowed service accounts to get roles from the realm-management client. This PR allows us to set roles by any client. This functionality was needed to be able to set the allow-app-sharing-role on the jupyterhub service account.

[Adam-D-Lewis]

flyby: tornado coroutine -> native coroutine. We don't need to use a tornado coroutine.

[Adam-D-Lewis]

make service account name a variable?

[Adam-D-Lewis]

flyby: tornado coroutine to native coroutine

[Adam-D-Lewis]

flyby: rename jupyterhub_client_id to jupyterhub_client_uuid so devs don't get confused with jupyterhub's client_id which is different.

[Adam-D-Lewis]

The local integration test seems to be failing for unrelated reasons.