CHANGELOG.rst

Changelog

0.3.1 (03-09-25)

Maintenance

• Mark project as inactive across README and docs; Update project classifier to Development Status :: 7 - Inactive (PR #87 by @pankajkoti)

0.3.0 (08-08-24)

Deprecations

• Deprecate the provider and proxy instantiations to upstream official Apache Airflow Databricks provider (PR #84 by @pankajkoti)

0.2.2 (16-04-24)

Bug fixes

• Prevent creation of duplicate jobs in Databricks (PR #76 by @Hang1225)

0.2.1 (03-04-24)

Bug fixes

• Support existing_cluster_id for DatabricksNotebookOperator when used from within DatabricksWorkflowTaskGroup (PR #73 by @Hang1225)

0.2.0 (20-03-24)

Feature

• Allow users to specify Databricks API version via an environment variable (PR #66 by @pankajkoti)

0.1.6 (23-02-24)

Bug fixes

• Bump pydantic to >=1.10.0 to be compatible with Airflow 2.7.0+ (PR #61 by @w0ut0)

0.1.5 (31-07-23)

Feature

• Add operator that supports all task types (PR #55 by @crong-k)

Bug fixes

• Limit Pydantic < 2.0.0 until Airflow resolves incompatibilities (PR #52 by @tatiana)

Enhancements

• Update query ID to hello world query (PR #56 by @jlaneve)

0.1.4 (23-06-16)

Bug fixes

• Fix repairing tasks declared in inner task groups (PR #49 by @tatiana)
• Fix copying dependencies from task groups to tasks inside intermediate task groups (PR #47 by @tatiana)

Enhancements

• Documentation improvements (PRs #43 and #44 by @jlaneve)

0.1.3 (23-04-27)

Enhancements

• Associate a DatabricksNotebookOperator to a DatabricksWorkflowTaskGroup even if there are up to three levels TaskGroups in between (issue #29)
• Support templating the field notebook_params of the DatabricksNotebookOperator (issue #33)
• Extend notebook_params of the DatabricksNotebookOperator with the values defined in DatabricksWorkflowTaskGroup (issue #33)
• Improve example DAGs (issue #29)
• Overall README improvements (pull requests #23 and #24)

0.1.1 (23-03-13)

Enhancements

• DatabricksWorkflowTaskGroup and DatabricksNotebookOperator log the Databricks Job URL (issue #20)
• README improvement (issue #21)

0.1.0 (2023-03-10)

Features

• DatabricksWorkflowTaskGroup: Airflow Task Group that allows users to create a Databricks Workflow.
• DatabricksNotebookOperator: Airflow Operator which abstracts a pre-existing Databricks Notebook. Can be used independently to run the Notebook, or within a Databricks Workflow Task Group.
• AstroDatabricksPlugin: An Airflow Plugin which is installed by the default. It allows users, by using the UI, to view a Databricks job and retry running it in case of failure.

Known limitations

• Cancelling the Airflow task does not stop its execution in Databricks (issue #1).
• Users should not click the buttons "repair all" or "repair single task" while the DAG/task is running (issue #2).