-
Notifications
You must be signed in to change notification settings - Fork 27
Description
Describe the bug
When connecting to a Databricks Standard workspace (without Unity Catalog) using the JDBC driver, the SHOW CATALOGS command returns spark_catalog as the catalog name. However, the Databricks UI displays this catalog as hive_metastore, and other Databricks tools (Python SQL Connector, REST APIs) do not recognize spark_catalog, causing integration failures.
This inconsistency breaks downstream integrations:
- Databricks UI shows
hive_metastore, notspark_catalog - Databricks Python SQL Connector does not recognize
spark_catalog - Databricks REST APIs return 422 errors when
spark_catalogis passed as the catalog parameter
To Reproduce
Steps to reproduce the behavior:
- Connect to a Databricks Standard workspace (without Unity Catalog enabled) using the JDBC driver
- Execute the SQL query:
SHOW CATALOGS - Observe that the returned catalog name in the catalog column is
spark_catalog - Try to use
spark_catalogwith other Databricks APIs or the Python SQL Connector - it will fail
Expected behavior
For Standard workspaces (non-Unity Catalog), SHOW CATALOGS should return hive_metastore as the catalog name to be consistent with:
- The Databricks UI
- The Databricks Python SQL Connector
- The Databricks REST APIs
Screenshots
N/A
Client side logs
When executing SHOW CATALOGS on a Standard workspace, the JDBC driver returns:
catalog: "spark_catalog"
We added debug logging in our application to confirm this:
DEBUG Databricks JDBC driver returned 'spark_catalog' - bug still exists, applying workaround to use 'hive_metastore'
Client Environment (please complete the following information):
- OS: macOS (darwin 24.6.0), also reproducible on Linux
- Java version: Java 17
- Java vendor: OpenJDK
- Driver Version: 3.0.7 (also reproducible in 2.6.x, 2.7.x, and all 3.0.x versions)
- BI Tool (if used): Custom application using Spring Boot + jOOQ (enterprise)
- BI Tool version (if applicable): N/A
Additional context
Workaround: We currently have a workaround in our application that replaces spark_catalog with hive_metastore.
Impact: This issue has been present since at least March 2024 and affects any application that:
- Builds data catalog browsers displaying catalog names to users
- Passes catalog names to other Databricks APIs (e.g., preview/sync endpoints)
- Integrates with the Databricks Python SQL Connector
Note: Premium workspaces with Unity Catalog enabled correctly return catalog names like main - this issue only affects Standard workspaces.