Skip to content

[BUG] [Databricks] Fusion seed fails with dbt1308 after ~200s when cluster cold-start exceeds connection timeout #1612

@sheryllancheta

Description

@sheryllancheta

Describe the bug
When running dbt seed (or any dbt invocation) against a non-serverless Databricks cluster that requires a cold start, dbt Fusion fails with dbt1308: context deadline exceeded before executing any SQL. The failure occurs during the initial connection ping that dbt performs to create the target schema. The hardcoded connection timeout of ~200 seconds is insufficient for clusters with cold-start times that exceed that window (in this case ~7 minutes). Retrying the job after manually pre-warming the cluster succeeds without any code or config changes, confirming the timeout as the sole cause.

What version of dbt Fusion is this bug in? (find out by running dbt --version)
dbt-fusion 2.0.0-preview.172

Is this a discrepancy between the dbt Fusion Engine and dbt Core? Check one.

  • YES
  • NO

In dbt-core with the Databricks adapter, connect_timeout is a configurable parameter in profiles.yml, allowing users to set a value appropriate for their cluster startup time. In dbt Fusion, no equivalent configuration option is exposed — the ~200s timeout appears to be hardcoded in the runtime, making it impossible to work around without pre-warming the cluster externally

To Reproduce

  1. Configure a dbt Fusion project targeting a non-serverless Databricks cluster with a cold-start time greater than 200 seconds.
  2. Allow the cluster to fully terminate/suspend (reach idle auto-termination).
  3. Trigger a dbt seed (or dbt run) invocation from dbt Cloud without manually pre-starting the cluster.
  4. Observe that dbt Fusion waits approximately 200 seconds on the initial connection ping, then fails with:
Error dbt1308: Failed to create schema '<schema>' in database '<database>' in remote
for seed.<project>.<node>: failed to ping database: databricks: execution error:
failed to execute query: context deadline exceeded

Expected behavior
Databricks seed/invocation should not fail in Fusion solely because the warehouse takes longer than ~200 seconds to start, especially when the same workflow succeeds on the non-Fusion engine.

Screenshots
N/A

Operating System and CPU Type (please complete the following information):

  • Environment: dbt Cloud, multi-tenant orchestration (Fusion engine running in managed containers)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions