Send per-update pipeline development only in dev mode and deprecate pipeline-level development by Sankalp-Mittal · Pull Request #6863 · databricks/cli

Sankalp-Mittal · 2026-09-28T11:57:04Z

Changes

bundle run / pipelines run send the per-update development: true parameter when presets.pipelines_development is enabled (defaulted by mode: development). false is never sent, so otherwise the pipeline-level value applies.
Setting development on a pipeline (YAML, target override, or Python) emits a deprecation warning pointing to mode: development. Under bundle validate --strict this warning fails validation.
When a Python mutator sets development: false on a pipeline in a development-mode target, an additional warning says the value is ignored by bundle run and how to opt out (presets.pipelines_development: false).
Deploy is unchanged.

Why

The pipeline-level development property is being deprecated in favor of the per-update parameter on StartUpdate.

Known differences for bundle run on a development-mode target, which now runs in development mode when:

the target was switched to mode: development without redeploying,
the pipeline was set to production outside the bundle, or
a Python mutator sets development: false on the pipeline.

Tests

Acceptance: bundle/run/development-mode, development-override, development-mode-changes, bundle/python/pipeline-development-deprecated, and bundle/python/pipeline-development-mutator.

This pull request and its description were written by Isaac.

Add a `--development [true|false]` flag to `bundle run` that maps to the per-update `development` parameter on the pipelines StartUpdate API. When the flag is not passed, it defaults from the target mode: true for a development-mode target, false elsewhere. The value is always force-sent so the per-update value is authoritative: the API otherwise falls back to the pipeline-level development property, and `omitempty` would drop a false value. Co-authored-by: Isaac <no-reply@databricks.com>

…t preset Development mode no longer sets the deprecated pipeline-level `development` property on deployed pipelines, and the `presets.pipelines_development` field is removed entirely. Dev-vs-prod is now decided per update by `bundle run --development` (defaulted from the target mode), so the per-update flag is the single source of truth. Removes the dev-mode auto-set in apply_target_mode.go, the preset application in apply_presets.go, and the PipelinesDevelopment field + its schema annotation. Updates the affected unit tests and the presets fixture. The generated jsonschema.json is left to the pending SDK downstream regen. Also adds changelog fragments for the --development flag and this removal. Co-authored-by: Isaac <no-reply@databricks.com>

Remove the --development flag from bundle run and instead infer a pipeline update's development mode from the bundle target mode in the shared runner, so both bundle run and pipelines run get the behavior through one code path. Development is force-sent so the resolved per-update value always wins over the deprecated pipeline-level property. Co-authored-by: Isaac <no-reply@databricks.com>

Add bundle/run/development-mode, which runs a pipeline under a production and a development target and asserts the update request force-sends development false and true respectively. Regenerate goldens affected by the feature: pipeline run/dry-run update bodies now carry development, and development-mode deploys no longer stamp development or the pipelines_development preset. Co-authored-by: Isaac <no-reply@databricks.com>

eng-dev-ecosystem-bot · 2026-09-28T12:35:36Z

Integration test report

Commit: 6c7a58d

Run: 37461667273

	Env	✅pass	🙈skip	Time
✅	aws linux	290	23	8:04
✅	aws windows	286	23	5:06
✅	azure linux	289	23	7:15
✅	azure windows	285	23	4:39
✅	gcp linux	290	23	7:05
✅	gcp windows	286	23	4:17

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
4:26	azure linux	TestAccept
3:59	aws linux	TestAccept
3:52	gcp linux	TestAccept
3:39	azure windows	TestAccept
2:50	gcp windows	TestAccept
2:43	aws windows	TestAccept

Regenerate the schema so it no longer lists the removed presets.pipelines_development setting, and drop placeholder annotations that an earlier regen on a pre-release SDK had left in annotations.yml. Co-authored-by: Isaac <no-reply@databricks.com>

- Cover `pipelines run` on a development target in the development-mode acceptance test. - Add development-override: an explicit pipeline-level development: true on a non-development target is passed through at deploy, but runs force-send development: false. - Make the runner test table-driven (no mode, development, production) and drop the redundant toPayload test. Co-authored-by: Isaac <no-reply@databricks.com>

…pement-flag-bundle-run

…elopment Restore main's deploy behavior (presets.pipelines_development, dev-mode stamping, schema). The pipeline runner now sends development: true only when the pipelines_development preset is enabled and omits the field otherwise, so the backend keeps using the pipeline-level property. Add a deprecation warning when the user sets development on a pipeline (YAML, target overrides, or Python). Co-authored-by: Isaac <no-reply@databricks.com>

Covers mode: development with pipeline development: false, switching the target mode without redeploying (both directions), and a pipeline switched to production outside the bundle. Co-authored-by: Isaac <no-reply@databricks.com>

Python mutators run after presets, so a mutator can set a dev-mode pipeline's development back to false; that false is deployed and main runs the pipeline in production. Only send the per-update development flag when the resolved pipeline value is also true, so this keeps working. Co-authored-by: Isaac <no-reply@databricks.com>

shreyas-goenka · 2026-09-30T15:02:01Z

@@ -0,0 +1,105 @@
+
+=== mode: development with pipeline development: false: deploy and run both use development: true


I don't follow, we have && r.pipeline.Development in the predicate, then why is development true being sent?

this is pre existing cli logic, if development:false and mode:development then the development property is set as true
Source Code

Thanks, in that case && r.pipeline.Development bit in the predicate here is redundant? Since we end up overriding it anyways here:

cli/bundle/config/mutator/resourcemutator/apply_presets.go

Line 102 in c232883

if config.IsExplicitlyEnabled(t.PipelinesDevelopment) {

the relevant predicate:

development := config.IsExplicitlyEnabled(r.bundle.Config.Presets.PipelinesDevelopment) && r.pipeline.Development req, err := opts.Pipeline.toPayload(r.pipeline, pipelineID, development)

Let's simplify it?

the code is actually not redundant, we can actually override r.pipeline.Development via python mutators which run after the apply_presets.go part runs, so removing it from the predicate breaks that behaviour, I had initially removed it but the python mutator case was caught

Regardless of what value we set here in PyDABs, the final by design is always meant to be the preset value right?

shreyas-goenka

The PR looks good, but please take a look at the one comment.

shreyas-goenka · 2026-10-02T13:33:44Z

@@ -0,0 +1,105 @@
+
+=== mode: development with pipeline development: false: deploy and run both use development: true


Thanks, in that case && r.pipeline.Development bit in the predicate here is redundant? Since we end up overriding it anyways here:

cli/bundle/config/mutator/resourcemutator/apply_presets.go

Line 102 in c232883

if config.IsExplicitlyEnabled(t.PipelinesDevelopment) {

the relevant predicate:

development := config.IsExplicitlyEnabled(r.bundle.Config.Presets.PipelinesDevelopment) && r.pipeline.Development req, err := opts.Pipeline.toPayload(r.pipeline, pipelineID, development)

Let's simplify it?

…pement-flag-bundle-run # Conflicts: # acceptance/bundle/deploy/snapshot-comparison/output.txt

Send development: true whenever the pipelines_development preset is on, as agreed in review. A Python mutator that sets development: false on a dev-mode pipeline now runs in development mode; the mutator acceptance test records this. Remove the runner unit test now covered by the development-mode acceptance test, and drop the terraform variant from the new tests' out.test.toml after the engine removal on main. Co-authored-by: Isaac <no-reply@databricks.com>

…ed mutator values Fix the wording of the deprecation warning. Add a second warning when a Python mutator sets development: false on a pipeline while the pipelines_development preset is on, since bundle run uses development mode from mode: development instead. Co-authored-by: Isaac <no-reply@databricks.com>

It warns on development values set by Python mutators, so it depends on running after them. Place it next to PythonMutator and say so. Co-authored-by: Isaac <no-reply@databricks.com>

## Release v1.20.0 ### Notable Changes * Remove the Terraform deployment engine. `bundle.engine: terraform` and `DATABRICKS_BUNDLE_ENGINE=terraform` now error, and a failed migration of existing Terraform state is reported as an error instead of falling back to Terraform. To keep deploying with Terraform, use Databricks CLI v1.19.x. ([#6888](#6888), [#6889](#6889)) ### CLI * `databricks aitools install` now supports Kiro, installing Databricks agent skills into its skills directory. ([#6908](#6908)) * Fixed `databricks api` corrupting integers larger than 2^53 (such as job and pipeline ids) — request bodies and responses now preserve them exactly. ([#6884](#6884)) * Added `--auth-mode` and `--set <plugin>.<resourceKey>.authMode=obo|sp|both` to `databricks apps init` so AppKit resources can be accessed on behalf of the user, by the service principal, or both. The default stays service principal. ([#6886](#6886)) * `databricks apps init` now requires a value for every field a service principal resource binding references, prompting for missing values in an interactive terminal and otherwise failing with the `--set` key to use, instead of creating a project with unset variables. ([#6903](#6903)) * Add `databricks apps init --package-manager <npm|pnpm>` to select the package manager for Node.js templates. Infer the default quietly from template lockfiles and AppKit version, check prerequisites before creating files, and preserve template formatting and pnpm version pins. ([#6902](#6902)) * Select npm or pnpm from `packageManager` declarations and lockfiles for `apps validate` and project validation during `apps deploy`. ([#6892](#6892)) * Fix `auth docker host` reporting the credential helper as configured when its executable is missing from `PATH`. ([#6880](#6880)) * Warn when the CLI binary was built more than 6 months ago and recommend updating. ([#6898](#6898)) ### AI Runtime * Add an experimental rank-partitioned container images to AI Runtime jobs. ([#6841](#6841)) * Support snapshot fields directly under `code_source` without requiring `type` or a nested `snapshot` block. ([#6927](#6927)) * Map AIR priority and Unity Catalog image fields when converting run configurations to bundles. ([#6905](#6905)) * Add workspace backend validation to `air run --dry-run`. ([#6934](#6934)) ### Bundles * Warn that `bundle.terraform` is deprecated and has no effect since the Terraform deployment engine was removed. ([#6940](#6940)) * Direct engine now detects and applies an explicitly configured zero-value boolean or float (e.g. `gcp_attributes.use_preemptible_executors: false`, `azure_attributes.spot_bid_max_price: 0`) added to a resource first deployed without the field, matching the existing handling of an explicit integer zero. ([#6882](#6882)) * Fix `bundle deployment migrate` failing with "no such file or directory" when the Terraform state has no resources or the configuration no longer declares any of them. ([#6958](#6958)) * `bundle run` and `pipelines run` now send the per-update `development` parameter for pipelines in development mode targets. Setting `development` on a pipeline is deprecated and now emits a warning; use `mode: development` instead. ([#6863](#6863)) * Remove the hidden `bundle debug terraform` command. ([#6933](#6933)) * Add support for `run_as.group_name` at the bundle and target levels for jobs and pipelines. ([#6676](#6676)) * Fix recreating a secret scope that was deleted outside of the bundle with the direct deployment engine. ([#6970](#6970)) * Accept title-case booleans (`True`/`False`, as rendered by Azure Pipelines) for boolean variables, and accept the same boolean strings (`yes`/`no`, `on`/`off`, ...) in Python bundles as in YAML. ([#6942](#6942)) ### Dependency Updates * Bump `github.com/databricks/databricks-sdk-go` from v0.182.0 to v0.185.0. ([#6928](#6928))

Sankalp-Mittal and others added 4 commits September 28, 2026 11:39

github-actions Bot added the DABs DABs related issues label Sep 28, 2026

Sankalp-Mittal and others added 4 commits September 28, 2026 13:49

Merge remote-tracking branch 'origin/main' into sankalp-mittal/develo…

ee1f55c

…pement-flag-bundle-run

github-actions Bot added the PyDABs label Sep 29, 2026

Sankalp-Mittal changed the title ~~Infer pipeline development mode per update from the bundle target mode~~ Send per-update pipeline development only in dev mode and deprecate pipeline-level development Sep 30, 2026

Sankalp-Mittal marked this pull request as ready for review September 30, 2026 14:37

Sankalp-Mittal requested review from a team as code owners September 30, 2026 14:37

Sankalp-Mittal requested review from andrewnester, denik, lennartkats-db and renaudhartert-db and removed request for andrewnester and renaudhartert-db September 30, 2026 14:37

shreyas-goenka reviewed Sep 30, 2026

View reviewed changes

remove redundant tests

7837414

shreyas-goenka reviewed Oct 2, 2026

View reviewed changes

shreyas-goenka approved these changes Oct 2, 2026

View reviewed changes

Sankalp-Mittal and others added 3 commits October 5, 2026 11:42

Merge branch 'main' into sankalp-mittal/developement-flag-bundle-run

06ea103

Merge remote-tracking branch 'origin/main' into sankalp-mittal/develo…

560e909

…pement-flag-bundle-run # Conflicts: # acceptance/bundle/deploy/snapshot-comparison/output.txt

Sankalp-Mittal enabled auto-merge October 6, 2026 11:12

Sankalp-Mittal disabled auto-merge October 6, 2026 11:12

shreyas-goenka approved these changes Oct 6, 2026

View reviewed changes

Comment thread bundle/phases/initialize.go Outdated

Run the pipeline development validator right after PythonMutator

6c7a58d

It warns on development values set by Python mutators, so it depends on running after them. Place it next to PythonMutator and say so. Co-authored-by: Isaac <no-reply@databricks.com>

Sankalp-Mittal enabled auto-merge October 6, 2026 12:12

Sankalp-Mittal added this pull request to the merge queue Oct 6, 2026

Merged via the queue into main with commit 419805b Oct 6, 2026
29 checks passed

Sankalp-Mittal deleted the sankalp-mittal/developement-flag-bundle-run branch October 6, 2026 12:40

This was referenced Oct 7, 2026

Update Databricks CLI to v1.20.0 databricks/databricks-vscode#2241

Closed

Update Databricks CLI to v1.20.0 and remove Terraform bundling databricks/databricks-vscode#2243

Merged

Navigation Menu

Repository navigation

Send per-update pipeline development only in dev mode and deprecate pipeline-level development - #6863

Send per-update pipeline development only in dev mode and deprecate pipeline-level development#6863
Sankalp-Mittal merged 16 commits into
mainfrom
sankalp-mittal/developement-flag-bundle-run

Sankalp-Mittal commented Sep 28, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

shreyas-goenka Sep 30, 2026

Uh oh!

Sankalp-Mittal Oct 1, 2026

Uh oh!

shreyas-goenka Oct 2, 2026

Uh oh!

Sankalp-Mittal Oct 2, 2026

Uh oh!

shreyas-goenka Oct 5, 2026

Uh oh!

shreyas-goenka left a comment

Uh oh!

shreyas-goenka Oct 2, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,105 @@

		=== mode: development with pipeline development: false: deploy and run both use development: true

Conversation

Sankalp-Mittal commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

Uh oh!

shreyas-goenka Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

Sankalp-Mittal Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

Sankalp-Mittal Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka left a comment

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sankalp-Mittal commented Sep 28, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Sep 28, 2026 •

edited

Loading