Query

type: "io.kestra.plugin.databricks.sql.Query"

Execute a SQL query on a Databricks cluster.

See Retrieve the connection details in the Databricks documentation to discover how to retrieve the needed configuration properties. We're using the Databricks JDBC driver to execute a Query, see https://docs.databricks.com/integrations/jdbc-odbc-bi.html#jdbc-driver-capabilities for its capabilities.

Due to current limitation of the JDBC driver with Java 21, Arrow is disabled, performance may be impacted, see here and here from Databricks status on Java 21 support.

Examples

id: databricks_sql_query
namespace: company.team

tasks:
  - id: sql_query
    type: io.kestra.plugin.databricks.sql.Query
    accessToken: <your-accessToken>
    host: <your-host>
    httpPath: <your-httpPath>
    sql: SELECT 1

Properties

host

• Type: string
• Dynamic: ✔️
• Required: ✔️

Databricks host.

httpPath

• Type: string
• Dynamic: ✔️
• Required: ✔️

Databricks cluster HTTP Path.

To retrieve the HTTP Path, go to your Databricks cluster, click on Advanced options then, click on JDBC/ODBC. See Retrieve the connection details for more details.

sql

• Type: string
• Dynamic: ✔️
• Required: ✔️

SQL query to be executed.

accessToken

• Type: string
• Dynamic: ✔️
• Required: ❌

Databricks access token.

catalog

• Type: string
• Dynamic: ✔️
• Required: ❌

properties

• Type: object
• SubType: string
• Dynamic: ❌
• Required: ❌

schema

• Type: string
• Dynamic: ✔️
• Required: ❌

timeZoneId

• Type: string
• Dynamic: ❌
• Required: ❌

The time zone id to use for date/time manipulation. Default value is the worker default zone id.

Outputs

size

• Type: integer
• Required: ❌

The number of fetched rows.

uri

• Type: string
• Required: ❌
• Format: uri

The URI of the result file in Kestra's internal storage (.ion file i.e. Amazon Ion text format).

Definitions

Metrics

fetch.size

• Type: counter