Skip to content

[ISSUE] Config inferring credentials from env #1183

@joshbrumpton-octopus

Description

@joshbrumpton-octopus

Description

The Config object fails validation when multiple authentication credentials are present in the environment, even when a specific auth method is explicitly configured via parameters.

When explicitly passing client_id and client_secret to the Config object, the SDK still scans environment variables and fails if DATABRICKS_TOKEN is also present, despite not being used.

The explicit auth configuration passed to Config should take precedence over environment variables, or there should be an option (e.g., auth_type) to restrict which credentials are considered.

  File "/home/circleci/project/.venv/lib/python3.11/site-packages/databricks/sdk/config.py", line 190, in __init__
    raise ValueError(message) from e
ValueError: validate: more than one authorization method configured: oauth and pat. Config: host=https://*****************************************, token=***, client_id=************************************, client_secret=***. Env: DATABRICKS_TOKEN, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET

Reproduction

import os
from databricks.sdk import Config

os.environ["DATABRICKS_TOKEN"] = "dapi_xxx"
os.environ["DATABRICKS_CLIENT_ID"] = "my-client-id"
os.environ["DATABRICKS_CLIENT_SECRET"] = "my-client-secret"

# Explicitly configure OAuth, but SDK fails due to DATABRICKS_TOKEN in env
config = Config(
    host="https://my-workspace.cloud.databricks.com",
    client_id=os.environ["DATABRICKS_CLIENT_ID"],
    client_secret=os.environ["DATABRICKS_CLIENT_SECRET"]
)

Expected behavior

When auth credentials are explicitly passed to Config, those should take precedence and the SDK should not fail due to unrelated environment variables.

Possible solutions:

  1. Explicit parameters override environment variables
  2. Add an auth_type parameter that restricts which credentials are considered
  3. Add an ignore_environment flag to disable env var scanning

Is it a regression?

Unknown — this may have been the intended behavior since unified auth was introduced.

Debug Logs

N/A — the error occurs during Config.__init__ before debug logging can be captured.

Other Information

  • OS: Linux (CircleCI)
  • Python: 3.11
  • databricks-sdk version: 0.47.0

Additional context

This is particularly problematic in CI/CD environments where multiple Databricks credentials may exist for different purposes (e.g., PAT for one tool, OAuth for another). The current behavior forces users to carefully unset environment variables, which is error-prone and unintuitive.

This issue also affects downstream projects like dbt-databricks, where users configure OAuth in profiles.yml but the adapter fails because an unrelated DATABRICKS_TOKEN exists in the environment.

SImilar issue: #680

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions