draft: feat: databricks connector

cli/cmd/runtime/start.go

@@ -35,6 +35,7 @@ import (
 	_ "github.com/rilldata/rill/runtime/drivers/bigquery"
 	_ "github.com/rilldata/rill/runtime/drivers/claude"
 	_ "github.com/rilldata/rill/runtime/drivers/clickhouse"
+	_ "github.com/rilldata/rill/runtime/drivers/databricks"
 	_ "github.com/rilldata/rill/runtime/drivers/druid"
 	_ "github.com/rilldata/rill/runtime/drivers/duckdb"
 	_ "github.com/rilldata/rill/runtime/drivers/file"

docs/docs/developers/build/connectors/connectors.md

@@ -95,6 +95,7 @@ Rill is continually evaluating additional OLAP engines that can be added. For a
 
 ### Athena
 ### BigQuery
+### Databricks
 ### Redshift
 ### Snowflake
 
@@ -115,7 +116,14 @@ Rill is continually evaluating additional OLAP engines that can be added. For a
     linkLabel="Learn more"
     referenceLink="bigquery"
   />
-
+  <ConnectorIcon
+    icon={<img src="/img/build/connectors/icons/Logo-Databricks.svg" alt="Databricks" className="duckdb-icon"/>}
+    header="Databricks"
+    content="Connect to Databricks SQL Warehouses for unified analytics powered by Unity Catalog."
+    link="/developers/build/connectors/data-source/databricks"
+    linkLabel="Learn more"
+    referenceLink="databricks"
+  />
   <ConnectorIcon
     icon={<img src="/img/build/connectors/icons/Logo-Redshift.svg" alt="Redshift" />}
     header="Redshift"

docs/docs/developers/build/connectors/data-source/data-source.md

@@ -40,6 +40,7 @@ Rill supports connecting your data to both [DuckDB](/developers/build/connectors
 
 ### Athena
 ### BigQuery
+### Databricks
 ### Redshift
 ### Snowflake
 
@@ -60,6 +61,14 @@ Rill supports connecting your data to both [DuckDB](/developers/build/connectors
     linkLabel="Learn more"
     referenceLink="bigquery"
   />
+  <ConnectorIcon
+    icon={<img src="/img/build/connectors/icons/Logo-Databricks.svg" alt="Databricks" className="duckdb-icon"/>}
+    header="Databricks"
+    content="Connect to Databricks SQL Warehouses for unified analytics powered by Unity Catalog."
+    link="/developers/build/connectors/data-source/databricks"
+    linkLabel="Learn more"
+    referenceLink="databricks"
+  />
 
   <ConnectorIcon
     icon={<img src="/img/build/connectors/icons/Logo-Redshift.svg" alt="Redshift" />}

docs/docs/developers/build/connectors/data-source/databricks.mdx

@@ -0,0 +1,138 @@
+---
+title: Databricks
+description: Connect to data in Databricks
+sidebar_label: Databricks
+sidebar_position: 15
+---
+
+import { TwoStepFlowIntro, EnvPullTip, DevProdSeparation, DeployToCloud } from '@site/src/components/connector';
+
+{/* WARNING: There are links to this page in source code. If you move it, find and replace the links and consider adding a redirect in docusaurus.config.js. */}
+
+## Overview
+
+[Databricks](https://docs.databricks.com/) is a unified analytics platform built on Apache Spark that provides a collaborative environment for data engineering, data science, and machine learning. With Unity Catalog, Databricks organizes data in a three-level hierarchy: **catalog > schema > table**. Rill connects to Databricks SQL Warehouses using the [Databricks SQL Go Driver](https://github.com/databricks/databricks-sql-go) and can be used both as a data source (to import data) and as an OLAP engine (to power dashboards directly).
+
+## Authentication
+
+To connect to Databricks, you need a personal access token and your SQL warehouse connection details.
+
+<TwoStepFlowIntro
+  connector="Databricks"
+  step1Description="Set up your Databricks connector with workspace hostname, HTTP path, and access token"
+  step2Description="Define which catalog, schema, table, or query to execute"
+/>
+
+### Using the UI
+
+1. Click **Add Data** in your Rill project
+2. Select **Databricks** as the data source type
+3. In the authentication step:
+   - Enter your workspace hostname
+   - Enter the HTTP path to your SQL warehouse
+   - Enter your personal access token
+   - Optionally specify a catalog and schema
+4. In the data model configuration step, enter your SQL query
+5. Click **Create** to finalize
+
+After the model YAML is generated, you can add additional [model settings](/developers/build/models/source-models) directly to the file.
+
+### Manual Configuration
+
+If you prefer to configure manually:
+
+**Step 1: Create connector configuration**
+
+Create `connectors/databricks.yaml`:
+
+```yaml
+type: connector
+driver: databricks
+
+host: adb-12345.azuredatabricks.net
+http_path: /sql/1.0/warehouses/abc123
+token: "{{ .env.DATABRICKS_TOKEN }}"
+catalog: main
+schema: default
+```
+
+**Step 2: Add credentials to `.env`**
+
+```bash
+DATABRICKS_TOKEN=dapi...
+```
+
+<EnvPullTip />
+
+Then, [create your first model](#create-your-first-model).
+
+## Create Your First Model
+
+Once your connector is configured, create a model to define what data to pull.
+
+Create `models/databricks_data.yaml`:
+
+```yaml
+type: model
+connector: databricks
+
+dev:
+  sql: SELECT * FROM my_catalog.my_schema.my_table LIMIT 10000
+
+sql: SELECT * FROM my_catalog.my_schema.my_table
+```
+
+After creating the model, you can add additional [model settings](/developers/build/models/source-models) directly to the file.
+
+## Finding Your Connection Details
+
+### Workspace Hostname
+
+Your workspace hostname is the URL you use to access Databricks, without the `https://` prefix. For example: `adb-12345678901234.azuredatabricks.net`.
+
+### HTTP Path
+
+To find the HTTP path for your SQL warehouse:
+
+1. In your Databricks workspace, navigate to **SQL Warehouses**
+2. Select your warehouse
+3. Go to the **Connection details** tab
+4. Copy the **HTTP path** value (e.g., `/sql/1.0/warehouses/abc123`)
+
+### Personal Access Token
+
+To generate a personal access token:
+
+1. In your Databricks workspace, click your username in the top bar
+2. Select **Settings**
+3. Go to **Developer** > **Access tokens**
+4. Click **Generate new token**
+5. Give it a description and set an expiration
+
+## Naming Mapping
+
+Databricks uses Unity Catalog's three-level hierarchy. In Rill's API:
+
+| Rill Parameter   | Databricks Concept | Example            |
+| ---------------- | ------------------ | ------------------ |
+| `database`       | Catalog            | `main`             |
+| `databaseSchema` | Schema             | `default`          |
+| `table`          | Table              | `my_table`         |
+
+## Separating Dev and Prod Environments
+
+<DevProdSeparation />
+
+## Deploy to Rill Cloud
+
+<DeployToCloud
+  connector="Databricks"
+  connectorId="databricks"
+  credentialDescription="a Databricks personal access token as the connector's `token` field"
+/>
+
+:::info Need help connecting to Databricks?
+
+If you would like to connect Rill to an existing Databricks instance, please don't hesitate to [contact us](/contact). We'd love to help!
+
+:::

docs/docs/reference/project-files/connectors.md

@@ -20,6 +20,7 @@ Connector YAML files define how Rill connects to external data sources and OLAP
 ### _Data Warehouses_
 - [**Athena**](#athena) - Amazon Athena
 - [**BigQuery**](#bigquery) - Google BigQuery
+- [**Databricks**](#databricks) - Databricks SQL Warehouse
 - [**Redshift**](#redshift) - Amazon Redshift
 - [**Snowflake**](#snowflake) - Snowflake data warehouse
 
@@ -312,6 +313,47 @@ ssl: true # Enable SSL for secure connection
 cluster: "mycluster" # Cluster name
 ```
 
+## Databricks
+
+### `driver`
+
+_[string]_ - Refers to the driver type and must be driver `databricks` _(required)_
+
+### `host`
+
+_[string]_ - Your Databricks workspace hostname (e.g., adb-12345.azuredatabricks.net) 
+
+### `http_path`
+
+_[string]_ - The HTTP path to your SQL warehouse or cluster (e.g., /sql/1.0/warehouses/abc123) 
+
+### `token`
+
+_[string]_ - A Databricks personal access token 
+
+### `catalog`
+
+_[string]_ - Unity Catalog catalog name. If not set, the workspace default is used 
+
+### `schema`
+
+_[string]_ - Default schema within the catalog 
+
+### `log_queries`
+
+_[boolean]_ - Controls whether to log raw SQL queries 
+
+```yaml
+# Example: Databricks connector configuration
+type: connector # Must be `connector` (required)
+driver: databricks # Must be `databricks` _(required)_
+host: "adb-12345.azuredatabricks.net" # Databricks workspace hostname
+http_path: "/sql/1.0/warehouses/abc123" # HTTP path to SQL warehouse
+token: "{{ .env.DATABRICKS_TOKEN }}" # Personal access token
+catalog: "main" # Unity Catalog catalog name
+schema: "default" # Default schema
+```
+
 ## Druid
 
 ### `driver`