Connect Databricks

Databricks SQL Warehouse, via the Statement Execution API. Dashies exchanges a service principal's OAuth credentials for a bearer token and runs statements over HTTPS.

Before you start

In your Databricks workspace:

• Create a service principal with the Databricks SQL and Workspace entitlements.
• Grant it CAN USE on the SQL warehouse Dashies will run against.
• Grant it read on the catalog and the schemas you want readable.
• Mint an OAuth machine-to-machine client secret for it. You need the client id and the secret.

You also need the SQL warehouse's id, which appears in its connection details in the Databricks UI.

1. Add the data source

Open dashies.xyz/app/connections, click to add a data source, and pick Databricks.

Field	Required	Example	Notes
Workspace host	yes	dbc-xxxxxxxx-xxxx.cloud.databricks.com	Host only, no scheme and no path.
Warehouse ID	yes	1234567890abcdef	Letters and digits, up to 64 characters.
Client ID	yes	a UUID	The application id of the service principal.
Catalog	no	main	Letters, digits, underscores; up to 128 characters.
Client secret	yes		20 to 256 characters.
Display name	no	Analytics warehouse	Up to 120 characters.

There is no schemas field. Databricks discovers the schemas during verification and stores up to 200 of them as the allowlist. To change what is readable, adjust the service principal's grants in Databricks and press Test.

BigQuery, Snowflake, and Redshift all derive their host from the fields above. Databricks is the one API-based engine that takes the host from you, and it is constrained by a vendor suffix allowlist rather than by resolving and range-checking the address. The host must end in .cloud.databricks.com, .azuredatabricks.net, or .gcp.databricks.com. An IP address in any form, an embedded port, a path, a scheme, or a user prefix are all refused with:

workspace_host is not a valid Databricks workspace host

2. It verifies in one step

Databricks is a single-step connect. Creating the data source exchanges the client credentials for a bearer token, lists the schemas the service principal can read, and comes back active with what it found, or error with a class. There is no Resync button on Databricks.

If it fails:

Message	Meaning
Databricks denied access. Check the client ID and secret, and that the service principal can use the SQL warehouse and read the catalog.	Authentication, or a missing grant.
We couldn't reach Databricks. Check the workspace host and try again.	Network-level failure reaching the host.
We couldn't read that workspace. Check the warehouse ID and catalog, then try again.	The token worked but the target was not usable.

Cost

Connecting is free. Verification uses the Unity Catalog REST API rather than a statement, so it starts no warehouse and burns no warehouse compute.

Introspection normally uses the same zero-warehouse path. It falls back to a information_schema.columns query, which does start the warehouse, only when the service principal is not authorized for the Unity Catalog REST API. If schema introspection is unexpectedly costing you compute, that fallback is why, and the fix is to grant the service principal Unity Catalog read access.

Caps

Limit	Value	What happens past it
Rows per dataset query	100,000	Refused before results are fetched.
Inline result envelope	about 25 MiB	The statement fails server-side.
Statement wait	30 seconds, then polling	Query errors if it never completes.
Data island, whole dashboard	8,388,608 bytes	Publish is refused.
Compiled dashboard body	5,242,880 bytes	Publish is refused. Usually binds first.

The envelope cap is a server-side Databricks limit, so a result that exceeds it fails rather than being truncated. Larger results page through presigned external links instead.

Databricks supports every materialization, including Parquet offload for a row-level dataset.

How these ceilings relate to each other, and which one binds first, is in sizes and ceilings.

Dialect notes

Cube SQL is Databricks SQL, which is Spark SQL. It is not a PostgreSQL dialect.

• Table references are three-level and backtick-quoted where quoting is needed: from `main`.`analytics`.`orders`. The built-in samples catalog (samples.nyctaxi.trips, samples.tpch.*) is useful for a first test with no seed table of your own.

• Bucket a date with date_trunc('MONTH', ts) or date_format(ts, 'yyyy-MM'):

  select date_trunc('MONTH', ordered_at) as month,
         sum(amount) as revenue
  from main.analytics.orders
  where ordered_at >= current_timestamp() - interval 12 months
  group by 1
  order by 1

• Relative window: current_timestamp() - interval 12 months. Conditional count: count_if(c).

• Databricks preserves an unquoted output alias exactly as written, unlike Postgres and Redshift. sum(amount) as revenue comes back revenue.

• A TIMESTAMP value arrives in the data island as an ISO-8601 UTC string with a T separator and a trailing Z, such as 2024-01-15T10:30:00.123Z. Bucket and format it in SQL rather than parsing the raw text. Big integers keep full precision, as strings.

• The cube must be a single read-only SELECT. Dashies enforces that; Databricks itself would happily run DML, so that guard is the only thing stopping it.

• An ARRAY, MAP, or STRUCT column has to be addressed into before it is usable, and explode multiplies rows. Nothing rejects that, so the cube runs and the numbers are simply wrong. Aggregate back to the grain you meant. See Verify your numbers.

Rotating the secret

Edit the data source and paste a new OAuth client secret under Client secret.

Check it worked

1. The data source reads active on dashies.xyz/app/connections, and lists the schemas it discovered.

2. Ask your AI tool to introspect it and run one query:

   Introspect my Databricks data source, then validate this cube SQL against it: select 1 as ok

3. Then author a dashboard against it.