Skip to content

air: flatten code_source configuration - #6927

Merged
caroline-db merged 2 commits into
mainfrom
air/code-source-flattening
Oct 5, 2026
Merged

caroline-db merged 2 commits into
mainfrom
air/code-source-flattening

Conversation

@caroline-db

@caroline-db caroline-db commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • Allow snapshot fields directly under code_source without requiring type or a nested snapshot block.
  • Preserve legacy configurations and overrides while normalizing supported forms to one internal path.
  • Hide legacy fields from help and reject mixed forms with actionable errors.

Why

The snapshot discriminator and wrapper are redundant now that snapshot is the only supported code source type. Flattening the configuration simplifies workload YAML and keeps future snapshot fields on one implementation path.

Tests

  • ./task checks
  • go test ./cmd/air
  • go test ./acceptance -run TestAccept/air/config-help -tail
  • go vet ./cmd/air
  • Built the PR CLI and successfully ran the AIR hello-world and a one-GPU Ray Core cookbook example with flattened code_source, including snapshot upload and extraction.
  • ./task test ran 11,580 tests; affected AIR tests passed, but an unrelated libs/dyn/jsonloader assertion fails on changed Go JSON error text.
  • Full lint is blocked by a golangci-lint/Staticcheck buildir panic in dependency package poll.

This PR was written by Codex.

@caroline-db
caroline-db requested review from a team as code owners October 5, 2026 14:14
@caroline-db
caroline-db requested a review from rclarey October 5, 2026 14:14
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Oct 5, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 3503d3c

Run: 37323506960

Env ✅​pass 🙈​skip Time
✅​ aws linux 276 17 6:08
✅​ aws windows 278 15 3:30
✅​ azure linux 275 17 6:14
✅​ azure windows 277 15 4:05
✅​ gcp linux 276 17 5:31
✅​ gcp windows 278 15 3:29
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:02 azure linux TestAccept
4:02 azure windows TestAccept
4:00 aws linux TestAccept
3:59 gcp linux TestAccept
3:28 aws windows TestAccept
3:27 gcp windows TestAccept

@caroline-db
caroline-db requested review from ben-hansen-db and maggiewang-db and removed request for rclarey October 5, 2026 14:56

@ben-hansen-db ben-hansen-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, can you test on one of our examples with built binary?

@@ -0,0 +1 @@
* Support snapshot fields directly under `code_source` without requiring `type` or a nested `snapshot` block. ([#6927](https://github.com/databricks/cli/pull/6927))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

@caroline-db
caroline-db added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026
@caroline-db
caroline-db added this pull request to the merge queue Oct 5, 2026
Merged via the queue into main with commit 7987c78 Oct 5, 2026
40 of 43 checks passed
@caroline-db
caroline-db deleted the air/code-source-flattening branch October 5, 2026 21:40
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 7987c78

Run: 37377458542

Env ❌​FAIL 🔄​flaky ✅​pass 🙈​skip Time
❌​ aws linux 7 1427 1120 119:01
❌​ aws windows 7 1352 1145 136:27
✅​ azure linux 1283 1172 103:41
🔄​ azure windows 6 1202 1197 119:46
✅​ gcp linux 1272 1176 115:04
🔄​ gcp windows 3 1194 1201 137:13
15 interesting tests: 8 flaky, 7 FAIL
Test Name aws linux aws windows azure windows gcp windows
❌​ TestAccept ❌​F ❌​F ✅​p 🔄​f
🔄​ TestAccept/bundle/invariant/delete_idempotent ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=app.yml.tmpl/READPLAN=1 ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/apps/lifecycle-started ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/apps/lifecycle-started/DMS=true ✅​p ✅​p ✅​p 🔄​f
🔄​ TestAccept/bundle/resources/cluster_policies/basic ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/cluster_policies/basic/DMS=true ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/clusters/deploy/data_security_mode ✅​p ✅​p 🔄​f ✅​p
🔄​ TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true ✅​p ✅​p 🔄​f ✅​p
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions ❌​F ❌​F 🙈​s 🙈​s
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DMS= ❌​F ❌​F
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DMS=true ❌​F ❌​F
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions ❌​F ❌​F 🙈​s 🙈​s
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DMS= ❌​F ❌​F
❌​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DMS=true ❌​F ❌​F
Top 50 slowest tests (at least 2 minutes):
duration env testname
11:28 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
10:53 gcp linux TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
10:46 gcp windows TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
10:12 gcp windows TestAccept/bundle/resources/apps/lifecycle-started/DMS=
10:09 gcp linux TestAccept/bundle/resources/apps/lifecycle-started/DMS=
10:09 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
8:57 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
8:48 gcp linux TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
8:42 gcp windows TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
8:29 aws linux TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:20 aws windows TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:10 azure linux TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
8:06 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:02 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
7:51 aws windows TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:39 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:27 gcp windows TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:20 aws linux TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:20 aws linux TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:11 azure linux TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:10 aws windows TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:08 aws linux TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:08 azure windows TestAccept/bundle/resources/apps/lifecycle-started/DMS=
7:06 azure windows TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
7:06 gcp windows TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:49 aws linux TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
6:49 gcp linux TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:47 aws linux TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:46 azure linux TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
6:43 gcp linux TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
6:35 azure windows TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:28 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
6:25 azure linux TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:20 azure linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
6:19 azure windows TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:19 aws windows TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
6:12 azure linux TestAccept/bundle/resources/apps/lifecycle-started/DMS=
5:55 aws windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:54 gcp linux TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:49 azure linux TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
5:48 azure windows TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:44 aws linux TestAccept/bundle/resources/clusters/deploy/simple/DMS=
5:10 aws windows TestAccept/bundle/deploy/spark-jar-task/DMS=
5:10 aws linux TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
4:57 azure windows TestAccept/bundle/deploy/spark-jar-task/DMS=true
4:56 azure windows TestAccept/bundle/deploy/spark-jar-task/DMS=
4:45 gcp linux TestAccept/bundle/deploy/spark-jar-task/DMS=true
4:21 gcp linux TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
4:20 azure linux TestAccept/bundle/deploy/spark-jar-task/DMS=
4:18 gcp linux TestAccept/bundle/resources/apps/inline_config/DMS=true

deco-sdk-tagging Bot added a commit that referenced this pull request Oct 7, 2026
## Release v1.20.0

### Notable Changes

 * Remove the Terraform deployment engine. `bundle.engine: terraform` and `DATABRICKS_BUNDLE_ENGINE=terraform` now error, and a failed migration of existing Terraform state is reported as an error instead of falling back to Terraform. To keep deploying with Terraform, use Databricks CLI v1.19.x. ([#6888](#6888), [#6889](#6889))

### CLI

 * `databricks aitools install` now supports Kiro, installing Databricks agent skills into its skills directory. ([#6908](#6908))
 * Fixed `databricks api` corrupting integers larger than 2^53 (such as job and pipeline ids) — request bodies and responses now preserve them exactly. ([#6884](#6884))
 * Added `--auth-mode` and `--set <plugin>.<resourceKey>.authMode=obo|sp|both` to `databricks apps init` so AppKit resources can be accessed on behalf of the user, by the service principal, or both. The default stays service principal. ([#6886](#6886))
 * `databricks apps init` now requires a value for every field a service principal resource binding references, prompting for missing values in an interactive terminal and otherwise failing with the `--set` key to use, instead of creating a project with unset variables. ([#6903](#6903))
 * Add `databricks apps init --package-manager <npm|pnpm>` to select the package manager for Node.js templates. Infer the default quietly from template lockfiles and AppKit version, check prerequisites before creating files, and preserve template formatting and pnpm version pins. ([#6902](#6902))
 * Select npm or pnpm from `packageManager` declarations and lockfiles for `apps validate` and project validation during `apps deploy`. ([#6892](#6892))
 * Fix `auth docker host` reporting the credential helper as configured when its executable is missing from `PATH`. ([#6880](#6880))
 * Warn when the CLI binary was built more than 6 months ago and recommend updating. ([#6898](#6898))

### AI Runtime

 * Add an experimental rank-partitioned container images to AI Runtime jobs. ([#6841](#6841))
 * Support snapshot fields directly under `code_source` without requiring `type` or a nested `snapshot` block. ([#6927](#6927))
 * Map AIR priority and Unity Catalog image fields when converting run configurations to bundles. ([#6905](#6905))
 * Add workspace backend validation to `air run --dry-run`. ([#6934](#6934))

### Bundles

 * Warn that `bundle.terraform` is deprecated and has no effect since the Terraform deployment engine was removed. ([#6940](#6940))
 * Direct engine now detects and applies an explicitly configured zero-value boolean or float (e.g. `gcp_attributes.use_preemptible_executors: false`, `azure_attributes.spot_bid_max_price: 0`) added to a resource first deployed without the field, matching the existing handling of an explicit integer zero. ([#6882](#6882))
 * Fix `bundle deployment migrate` failing with "no such file or directory" when the Terraform state has no resources or the configuration no longer declares any of them. ([#6958](#6958))
 * `bundle run` and `pipelines run` now send the per-update `development` parameter for pipelines in development mode targets. Setting `development` on a pipeline is deprecated and now emits a warning; use `mode: development` instead. ([#6863](#6863))
 * Remove the hidden `bundle debug terraform` command. ([#6933](#6933))
 * Add support for `run_as.group_name` at the bundle and target levels for jobs and pipelines. ([#6676](#6676))
 * Fix recreating a secret scope that was deleted outside of the bundle with the direct deployment engine. ([#6970](#6970))
 * Accept title-case booleans (`True`/`False`, as rendered by Azure Pipelines) for boolean variables, and accept the same boolean strings (`yes`/`no`, `on`/`off`, ...) in Python bundles as in YAML. ([#6942](#6942))

### Dependency Updates

 * Bump `github.com/databricks/databricks-sdk-go` from v0.182.0 to v0.185.0. ([#6928](#6928))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants