bundle: reject the terraform deployment engine by denik · Pull Request #6888 · databricks/cli

denik · 2026-09-30T11:03:30Z

engine: terraform (bundle config) and DATABRICKS_BUNDLE_ENGINE=terraform now error instead of running the terraform engine: the message points users to remove the setting (existing terraform state migrates to the direct engine automatically) or install Databricks CLI v1.19.x. The tf->direct migration that runs before deploy is now mandatory - it hard-errors on a parse/convert/plan-check failure instead of silently falling back to terraform, since there is no engine left to fall back to. This is the removal announced in #6765.

The terraform engine code is still present and is deleted in a follow-up (#6889); this change only makes it unreachable. Acceptance tests that drove the terraform engine are adapted (engine-config rejection, migrate plan-check failure now fatal, state_present replays a fixture instead of deploying on terraform, empty_string_dropped drops the cross-engine comparison).

This pull request and its description were written by Isaac.

eng-dev-ecosystem-bot · 2026-09-30T11:54:46Z

Integration test report

Commit: 57da9c7

Run: 37311642030

	Env	✅pass	🙈skip	Time
✅	aws linux	285	35	6:06
✅	aws windows	284	34	4:09
✅	azure linux	284	35	6:23
✅	azure windows	283	34	4:32
✅	gcp linux	285	35	5:39
✅	gcp windows	284	34	3:36

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
4:20	azure windows	TestAccept
4:01	aws linux	TestAccept
4:01	azure linux	TestAccept
3:57	aws windows	TestAccept
3:15	gcp windows	TestAccept
3:00	gcp linux	TestAccept

…#6877) Removes the terraform engine from the acceptance CI matrix so the suite runs direct-only, and drops the now-single-value `DATABRICKS_BUNDLE_ENGINE` matrix pins (using `Env.DATABRICKS_BUNDLE_ENGINE` only where a test forces a specific engine). The terraform engine code itself is untouched here; it is rejected in [databricks#6888](databricks#6888) and removed in [databricks#6889](databricks#6889). This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

Selecting the terraform engine now errors instead of running it: ResolveEngineSetting and the validate:engine mutator reject `engine: terraform` (config) and DATABRICKS_BUNDLE_ENGINE=terraform with engine.TerraformRemovedMessage, pointing users to remove the setting (migration is automatic) or install CLI v1.18.x. The tf->direct migration is now mandatory: statemgmt.Migrate hard-errors on parse/convert/plan-check failure (with migrationFailedHint) instead of silently falling back to the terraform engine, since there is no engine left to fall back to. The terraform engine code is still present (deleted in a follow-up); this change only makes it unreachable. Co-authored-by: Isaac <no-reply@databricks.com>

With `engine: terraform` / DATABRICKS_BUNDLE_ENGINE=terraform now rejected and the tf->direct migration mandatory, adapt the tests that exercised the terraform engine: - migrate/command/engine-config-terraform, validate/invalid-engine-{bundle,target}: goldens now show the removal error / updated "expected direct" message. - validate/engine-config-valid: the terraform-target now fails validation (musterr). - migrate/auto/plan-failure: a failed migration plan check is now a hard error (no terraform fallback); the retry still succeeds and migrates. - empty_string_dropped: drop the terraform half of the cross-engine "" comparison (terraform dropped "" via omitempty); record only the direct engine now. Co-authored-by: Isaac <no-reply@databricks.com>

… notable-changes Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

…t-comparison The terraform-forcing tests now fail with the removal error; wrap the failing command in musterr so the expected non-zero exit is asserted (config-remote-sync -o json stays on errcode: it reports the error in the JSON payload with exit 0). snapshot-comparison compared a terraform-deployed snapshot against the direct one, which is no longer possible; migration compatibility is covered by the migrate command and invariant tests. Co-authored-by: Isaac <no-reply@databricks.com>

…st comments Co-authored-by: Isaac <no-reply@databricks.com>

…ntry Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Jan N Rose <janniklas.rose@gmail.com>

The golden already records the serial, and the substring check also matched larger serials. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com>

Split TerraformRemovedMessage into a summary and a detail so the bundle.engine validator and ResolveEngineSetting print the same text. Regenerate goldens for that and for the reworded messages from review. Co-authored-by: Isaac <no-reply@databricks.com>

Replace both terraform deploys with replay_tfstate.py of fixtures captured from a terraform run on main. Bundle 1 migrates with bundle deployment migrate, bundle 2 migrates automatically on deploy, and the test checks that both produce the same direct state. Co-authored-by: Isaac <no-reply@databricks.com>

…rm fixtures" This reverts commit 4eed848.

… state Replay a terraform deployment, change the job remotely, and check that config-remote-sync migrates the state in memory, detects the change, leaves terraform.tfstate in place, and saves the change into the config. Co-authored-by: Isaac <no-reply@databricks.com>

Upstream rewrote the test (#6888); its lineage now comes from the replayed tfstate fixture, so the hash is stable and nothing prints the post-destroy lineage. Drop the read_lineage.py call. Co-authored-by: Isaac <no-reply@databricks.com>

Deletes the now-unreachable Terraform deployment engine code. The previous PR ([databricks#6888](databricks#6888)) made `engine: terraform` error and the tf->direct migration mandatory, so nothing reaches the engine anymore. This is the removal announced in [databricks#6765](databricks#6765). Removed: the terraform engine implementation (apply/convert/plan/init/install/import/interpolate/pubkey/showplanfile/unbind/write) and the entire `tfdyn` package; the terraform branches in the deploy/bind/destroy phases; `Bundle.Terraform*` fields and `deployplan.NewPlanTerraform`; the `terraform-exec`, `hc-install` and `terraform-json` dependencies; and `terraform` from the `bundle.engine` schema enum. Also removed: the terraform-only YAML-sync config snapshot (`resources-config-sync-snapshot.json`), which only a terraform deploy wrote and only a terraform state read; the `snapshot-comparison` test that compared it with migrate output is dropped in [databricks#6888](databricks#6888). Kept: the state-migration infrastructure (terraform state parsing, and the state-mapping helpers still used by the migration, kept in `showplanfile.go`, plus `BindOptions` in `import.go`) so existing Terraform state still migrates to the direct engine. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

…cks#6930) ## Why To record current behaviour. There was at least one user that used ${databricks_x..} reference. ## Changes Adds `migrate/auto/tf-resource-type-references`: replay a terraform deployment whose job references its pipeline as `${databricks_pipeline.src_pipeline.id}` (fixture captured from a terraform deploy just before [databricks#6888](databricks#6888), where terraform resolved it), then auto-migrate. The direct engine does not support terraform resource-type references (only terraform field paths inside `${resources.*}`, [databricks#5392](databricks#5392)): `validate` passes, then plan and deploy fail with `invalid dependency`. The test records this with `musterr`, so bundles using this syntax are stuck after the terraform removal until it is supported or rejected with an actionable error. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

## Release v1.20.0 ### Notable Changes * Remove the Terraform deployment engine. `bundle.engine: terraform` and `DATABRICKS_BUNDLE_ENGINE=terraform` now error, and a failed migration of existing Terraform state is reported as an error instead of falling back to Terraform. To keep deploying with Terraform, use Databricks CLI v1.19.x. ([#6888](#6888), [#6889](#6889)) ### CLI * `databricks aitools install` now supports Kiro, installing Databricks agent skills into its skills directory. ([#6908](#6908)) * Fixed `databricks api` corrupting integers larger than 2^53 (such as job and pipeline ids) — request bodies and responses now preserve them exactly. ([#6884](#6884)) * Added `--auth-mode` and `--set <plugin>.<resourceKey>.authMode=obo|sp|both` to `databricks apps init` so AppKit resources can be accessed on behalf of the user, by the service principal, or both. The default stays service principal. ([#6886](#6886)) * `databricks apps init` now requires a value for every field a service principal resource binding references, prompting for missing values in an interactive terminal and otherwise failing with the `--set` key to use, instead of creating a project with unset variables. ([#6903](#6903)) * Add `databricks apps init --package-manager <npm|pnpm>` to select the package manager for Node.js templates. Infer the default quietly from template lockfiles and AppKit version, check prerequisites before creating files, and preserve template formatting and pnpm version pins. ([#6902](#6902)) * Select npm or pnpm from `packageManager` declarations and lockfiles for `apps validate` and project validation during `apps deploy`. ([#6892](#6892)) * Fix `auth docker host` reporting the credential helper as configured when its executable is missing from `PATH`. ([#6880](#6880)) * Warn when the CLI binary was built more than 6 months ago and recommend updating. ([#6898](#6898)) ### AI Runtime * Add an experimental rank-partitioned container images to AI Runtime jobs. ([#6841](#6841)) * Support snapshot fields directly under `code_source` without requiring `type` or a nested `snapshot` block. ([#6927](#6927)) * Map AIR priority and Unity Catalog image fields when converting run configurations to bundles. ([#6905](#6905)) * Add workspace backend validation to `air run --dry-run`. ([#6934](#6934)) ### Bundles * Warn that `bundle.terraform` is deprecated and has no effect since the Terraform deployment engine was removed. ([#6940](#6940)) * Direct engine now detects and applies an explicitly configured zero-value boolean or float (e.g. `gcp_attributes.use_preemptible_executors: false`, `azure_attributes.spot_bid_max_price: 0`) added to a resource first deployed without the field, matching the existing handling of an explicit integer zero. ([#6882](#6882)) * Fix `bundle deployment migrate` failing with "no such file or directory" when the Terraform state has no resources or the configuration no longer declares any of them. ([#6958](#6958)) * `bundle run` and `pipelines run` now send the per-update `development` parameter for pipelines in development mode targets. Setting `development` on a pipeline is deprecated and now emits a warning; use `mode: development` instead. ([#6863](#6863)) * Remove the hidden `bundle debug terraform` command. ([#6933](#6933)) * Add support for `run_as.group_name` at the bundle and target levels for jobs and pipelines. ([#6676](#6676)) * Fix recreating a secret scope that was deleted outside of the bundle with the direct deployment engine. ([#6970](#6970)) * Accept title-case booleans (`True`/`False`, as rendered by Azure Pipelines) for boolean variables, and accept the same boolean strings (`yes`/`no`, `on`/`off`, ...) in Python bundles as in YAML. ([#6942](#6942)) ### Dependency Updates * Bump `github.com/databricks/databricks-sdk-go` from v0.182.0 to v0.185.0. ([#6928](#6928))

denik force-pushed the denik/reject-terraform branch from 55845d2 to dc5f96a Compare September 30, 2026 11:19

denik mentioned this pull request Sep 30, 2026

Remove the Terraform deployment engine #6864

Closed

2 of 4 tasks

denik force-pushed the denik/remove-tf-ci-matrix branch from b769255 to c9565df Compare October 1, 2026 13:42

denik force-pushed the denik/reject-terraform branch 2 times, most recently from 26b45a6 to cfa9047 Compare October 1, 2026 21:11

This was referenced Oct 2, 2026

bundle: delete the terraform deployment engine code #6889

Merged

Remove the terraform engine from the acceptance CI matrix #6877

Merged

denik force-pushed the denik/remove-tf-ci-matrix branch from c9565df to ac061d5 Compare October 2, 2026 09:34

denik force-pushed the denik/reject-terraform branch from cfa9047 to 3fd66af Compare October 2, 2026 09:34

denik force-pushed the denik/remove-tf-ci-matrix branch from 2d11a48 to aa4e242 Compare October 2, 2026 10:23

denik force-pushed the denik/reject-terraform branch from 3fd66af to 8e85239 Compare October 2, 2026 10:28

Base automatically changed from denik/remove-tf-ci-matrix to main October 2, 2026 11:04

denik force-pushed the denik/reject-terraform branch from 8e85239 to daa60dd Compare October 2, 2026 11:11

github-actions Bot added the DABs DABs related issues label Oct 2, 2026

denik force-pushed the denik/reject-terraform branch from daa60dd to 6db050d Compare October 2, 2026 11:44

denik force-pushed the denik/reject-terraform branch from 6db050d to 33ec5c0 Compare October 5, 2026 07:28

denik marked this pull request as ready for review October 5, 2026 07:33

denik requested review from a team, jefferycheng1, kanterov and lennartkats-db as code owners October 5, 2026 07:33

denik force-pushed the denik/reject-terraform branch from 95b6da3 to c5674aa Compare October 5, 2026 09:13

janniklasrose reviewed Oct 5, 2026

View reviewed changes

Comment thread acceptance/bundle/state/state_present/script Outdated

Comment thread bundle/statemgmt/direct_migration.go Outdated

Comment thread bundle/statemgmt/direct_migration.go

denik requested a review from janniklasrose October 5, 2026 12:12

pietern reviewed Oct 5, 2026

View reviewed changes

Comment thread bundle/config/engine/engine.go Outdated

Comment thread bundle/config/engine/engine.go Outdated

Comment thread bundle/config/validate/validate_engine.go Outdated

Comment thread bundle/statemgmt/direct_migration.go Outdated

denik and others added 2 commits October 5, 2026 14:26

denik and others added 11 commits October 5, 2026 14:26

bundle: terraform removed in v1.20.0 (keep via v1.19.x); changelog to…

276d704

… notable-changes Co-authored-by: Isaac <no-reply@databricks.com>

changelog: keep-Terraform version is v1.19.x

cc77f02

Co-authored-by: Isaac <no-reply@databricks.com>

acceptance: regenerate terraform-rejection goldens at the reject commit

96fdf36

Co-authored-by: Isaac <no-reply@databricks.com>

acceptance: describe the terraform rejection in config-remote-sync te…

b7e5033

…st comments Co-authored-by: Isaac <no-reply@databricks.com>

changelog: describe what changes for users in the terraform removal e…

4b9a52f

…ntry Co-authored-by: Isaac <no-reply@databricks.com>

Apply suggestion from @janniklasrose

896f516

Co-authored-by: Jan N Rose <janniklas.rose@gmail.com>

acceptance: drop contains.py on the state_present serial check

bdb66c9

The golden already records the serial, and the substring check also matched larger serials. Co-authored-by: Isaac <no-reply@databricks.com>

acceptance: update plan-failure golden for the reworded migration hint

79231f4

Co-authored-by: Isaac <no-reply@databricks.com>

Apply batched suggestions from code review

7d70fcd

Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com>

janniklasrose approved these changes Oct 5, 2026

View reviewed changes

denik force-pushed the denik/reject-terraform branch from 10eed7f to 4eed848 Compare October 5, 2026 12:38

denik and others added 2 commits October 5, 2026 14:42

Revert "acceptance: restore snapshot-comparison with replayed terrafo…

e9b19f7

…rm fixtures" This reverts commit 4eed848.

denik requested review from ilyakuz-db and pietern October 5, 2026 13:09

ilyakuz-db approved these changes Oct 5, 2026

View reviewed changes

Comment thread acceptance/bundle/config-remote-sync/terraform_state/out.test.toml

denik added this pull request to the merge queue Oct 5, 2026

Merged via the queue into main with commit 15f04e8 Oct 5, 2026
28 checks passed

denik deleted the denik/reject-terraform branch October 5, 2026 13:33

denik mentioned this pull request Oct 5, 2026

acc: terraform resource-type references after auto-migration #6930

Merged

This was referenced Oct 7, 2026

Update Databricks CLI to v1.20.0 databricks/databricks-vscode#2241

Closed

Update Databricks CLI to v1.20.0 and remove Terraform bundling databricks/databricks-vscode#2243

Merged

Navigation Menu

Repository navigation

bundle: reject the terraform deployment engine - #6888

bundle: reject the terraform deployment engine#6888
denik merged 19 commits into
mainfrom
denik/reject-terraform

denik commented Sep 30, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

denik commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

denik commented Sep 30, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Sep 30, 2026 •

edited

Loading