Skip to content

internal: Refactor immutable folder upload as an internal resource in direct engine #6084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrewnester merged 51 commits into from
Sep 8, 2026
Merged

internal: Refactor immutable folder upload as an internal resource in direct engine #6084

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
84608d6
internal: Refactor immutable folder upload as an internal resource in…
andrewnester Jul 28, 2026
a5ab287
fix test and tidy up
andrewnester Jul 28, 2026
4a0d4bd
updated the output
andrewnester Jul 28, 2026
c869f1d
Merge branch 'main' into feat/upload-resource
andrewnester Jul 28, 2026
aa73e34
fix lint
andrewnester Jul 28, 2026
b4f272f
make snapshot a normal resource
andrewnester Jul 31, 2026
f31a466
clean up
andrewnester Jul 31, 2026
5943979
fix test
andrewnester Aug 3, 2026
195974d
fixes
andrewnester Aug 3, 2026
b6bb79f
refschema
andrewnester Aug 3, 2026
2b7b48f
lint fixes
andrewnester Aug 3, 2026
54a7dc0
use HasInternalResources
andrewnester Aug 3, 2026
fbd5229
fix test and etc
andrewnester Aug 4, 2026
7825722
stable bundle id
andrewnester Aug 4, 2026
b1e0be1
fix tests
andrewnester Aug 4, 2026
b8cdcec
fix typo
andrewnester Aug 5, 2026
f32d7fc
update refschema
andrewnester Aug 5, 2026
94eefc7
remove deadcode + update test output
andrewnester Aug 5, 2026
769c0c3
adr feedback
andrewnester Aug 5, 2026
c5cbf31
refschema
andrewnester Aug 5, 2026
af7f3ab
fix invariant coverage test
andrewnester Aug 6, 2026
5a05041
fix fmt
andrewnester Aug 6, 2026
6c74b0b
Merge branch 'main' into feat/upload-resource
andrewnester Aug 6, 2026
e816436
fmt
andrewnester Aug 10, 2026
0c8c6b6
Merge branch 'main' into feat/upload-resource
andrewnester Aug 13, 2026
03d79bd
update out.test.toml
andrewnester Aug 13, 2026
d879631
Merge branch 'main' into feat/upload-resource
andrewnester Aug 18, 2026
ba0d5b8
fix build
andrewnester Aug 18, 2026
4ac3aed
fix lint
andrewnester Aug 18, 2026
d07f4f8
fix output
andrewnester Aug 18, 2026
2408d9a
fix delete count
andrewnester Aug 18, 2026
8f2b500
fixes
andrewnester Aug 19, 2026
2060f2b
Merge branch 'main' into feat/upload-resource
andrewnester Aug 19, 2026
82f2714
updated output
andrewnester Aug 20, 2026
92ec65c
fix output
andrewnester Aug 20, 2026
b60469f
build and store zip locally
andrewnester Aug 24, 2026
41a61a0
Merge branch 'main' into feat/upload-resource
andrewnester Aug 25, 2026
25ecb17
fixes
andrewnester Aug 25, 2026
f77c4f0
Merge branch 'main' into feat/upload-resource
andrewnester Sep 2, 2026
574bb0d
fixes
andrewnester Sep 2, 2026
9e512d2
updated test output
andrewnester Sep 2, 2026
2a92d18
fix fmt
andrewnester Sep 2, 2026
f171a94
fix lint and unit test
andrewnester Sep 2, 2026
1758215
fix unit test
andrewnester Sep 2, 2026
f1f4913
Merge branch 'main' into feat/upload-resource
andrewnester Sep 7, 2026
0805262
skip remote check for immutable folder
andrewnester Sep 7, 2026
fe8c7c0
revert test
andrewnester Sep 7, 2026
b09129b
no validate remote for immutable folder
andrewnester Sep 7, 2026
777d4ed
remove ensure remote check
andrewnester Sep 7, 2026
498f74a
remove unnecessary check
andrewnester Sep 8, 2026
3ad3160
fix cleanup
andrewnester Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions acceptance/bundle/deploy/immutable-no-artifacts/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*.txt
*.json
1 change: 1 addition & 0 deletions acceptance/bundle/deploy/immutable-no-artifacts/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

109 changes: 101 additions & 8 deletions acceptance/bundle/deploy/immutable-no-artifacts/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,113 @@ Workspace:

Validation OK!

>>> [CLI] bundle plan -o json

>>> jq .plan["resources.jobs.my_job"].new_state.value.tasks plan.json
[
{
"notebook_task": {
"base_parameters": {
"path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/some_path"
},
"notebook_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/notebook"
},
"task_key": "notebook_task"
},
{
"environment_key": "env",
"spark_python_task": {
"python_file": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/main.py"
},
"task_key": "spark_python_task"
}
]

>>> jq .plan["resources.internal_immutable_snapshots.immutable"] plan.json
{
"action": "create",
"new_state": {
"value": {
"relative_path": "[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"full_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"bundle_id": "[SNAPSHOT_HASH]",
"acl": [
{
"user_name": "[USERNAME]",
"permission_level": "CAN_READ"
}
],
"zip_path": "[TEST_TMP_DIR]/.databricks/bundle/default/snapshots/[SNAPSHOT_HASH].zip"
}
}
}

>>> [CLI] bundle deploy
Uploading immutable bundle snapshot...
Created internal_immutable_snapshots.immutable

@pietern pietern Aug 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should show up differently in the deploy output.

Created jobs.my_job
Files: 0 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
Resources: 2 created, 0 changed, 0 deleted, 0 unchanged

>>> jq .settings.tasks job.json
"/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/main.py"

>>> jq .settings.tasks job.json
"/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/notebook"

>>> [CLI] jobs get [NUMID]
"/Workspace/Users/[UUID]/.snapshots/[UUID]/[SNAPSHOT_HASH]/files/src/main.py"
>>> jq .settings.tasks job.json
"/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/some_path"

>>> [CLI] jobs get [NUMID]
"/Workspace/Users/[UUID]/.snapshots/[UUID]/[SNAPSHOT_HASH]/files/src/notebook"
>>> title Check snapshot plan after deploy:

>>> [CLI] jobs get [NUMID]
"/Workspace/Users/[UUID]/.snapshots/[UUID]/[SNAPSHOT_HASH]/files/some_path"
=== Check snapshot plan after deploy:
>>> [CLI] bundle plan -o json

>>> jq .plan["resources.internal_immutable_snapshots.immutable"] plan.json
{
"action": "skip",
"remote_state": {
"relative_path": "[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"full_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]"
},
"changes": {
"acl": {
"action": "skip",
"reason": "missing_in_remote",
"old": [
{
"user_name": "[USERNAME]",
"permission_level": "CAN_READ"
}
],
"new": [
{
"user_name": "[USERNAME]",
"permission_level": "CAN_READ"
}
],
"remote": null
},
"bundle_id": {
"action": "skip",
"reason": "missing_in_remote",
"old": "[SNAPSHOT_HASH]",
"new": "[SNAPSHOT_HASH]",
"remote": ""
},
"zip_path": {
"action": "skip",
"reason": "missing_in_remote",
"old": "[TEST_TMP_DIR]/.databricks/bundle/default/snapshots/[SNAPSHOT_HASH].zip",
"new": "[TEST_TMP_DIR]/.databricks/bundle/default/snapshots/[SNAPSHOT_HASH].zip",
"remote": ""
}
}
}

>>> [CLI] bundle deploy
Files: 0 uploaded, 0 deleted
Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
Snapshot did not change as expected

>>> [CLI] bundle destroy --auto-approve
The following resources will be deleted:
Expand Down
28 changes: 24 additions & 4 deletions acceptance/bundle/deploy/immutable-no-artifacts/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,33 @@ cleanup() {
trap cleanup EXIT

trace $CLI bundle validate
trace $CLI bundle deploy
trace $CLI bundle plan -o json > plan.json
trace jq '.plan["resources.jobs.my_job"].new_state.value.tasks' plan.json
trace jq '.plan["resources.internal_immutable_snapshots.immutable"]' plan.json

trace $CLI bundle deploy $(readplanarg plan.json)

# Get a job and check that task paths point into the snapshot
JOB_ID=$($CLI bundle summary -o json | jq -r '.resources.jobs.my_job.id')
trace $CLI jobs get $JOB_ID | jq '.settings.tasks' | jq '.[] | select(.spark_python_task != null) | .spark_python_task.python_file'
trace $CLI jobs get $JOB_ID | jq '.settings.tasks' | jq '.[] | select(.notebook_task != null) | .notebook_task.notebook_path'
trace $CLI jobs get $JOB_ID | jq '.settings.tasks' | jq '.[] | select(.notebook_task != null) | .notebook_task.base_parameters.path'
$CLI jobs get $JOB_ID > job.json
trace jq '.settings.tasks' job.json | jq '.[] | select(.spark_python_task != null) | .spark_python_task.python_file'
trace jq '.settings.tasks' job.json | jq '.[] | select(.notebook_task != null) | .notebook_task.notebook_path'
trace jq '.settings.tasks' job.json | jq '.[] | select(.notebook_task != null) | .notebook_task.base_parameters.path'
PYTHON_FILE_1=$(jq '.settings.tasks' job.json | jq '.[] | select(.spark_python_task != null) | .spark_python_task.python_file')

trace title "Check snapshot plan after deploy: "
trace $CLI bundle plan -o json > plan.json
trace jq '.plan["resources.internal_immutable_snapshots.immutable"]' plan.json

trace $CLI bundle deploy $(readplanarg plan.json)
rm plan.json
rm job.json

PYTHON_FILE_2=$($CLI jobs get $JOB_ID | jq '.settings.tasks' | jq '.[] | select(.spark_python_task != null) | .spark_python_task.python_file')
if [ "$PYTHON_FILE_1" == "$PYTHON_FILE_2" ]; then
echo "Snapshot did not change as expected"
else
echo "Snapshot changed"
exit 1
fi
trace $CLI bundle destroy --auto-approve
7 changes: 7 additions & 0 deletions acceptance/bundle/deploy/immutable-no-artifacts/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
Cloud = false # Temporary disable cloud tests until the API is fully available
RecordRequests = true
EnvMatrix.READPLAN = ["", "1"]

# immutable_folder only works with the direct engine.
EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]
Expand All @@ -17,3 +18,9 @@ Ignore = [
[[Repls]]
Old = '[0-9a-f]{64}'
New = '[SNAPSHOT_HASH]'

# When READPLAN=1, "bundle deploy" is called as "bundle deploy --plan plan.json".
# Normalize so both variants produce identical output.
[[Repls]]
Old = ' --plan plan.json'
New = ''
12 changes: 6 additions & 6 deletions acceptance/bundle/deploy/immutable-permissions-change/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@

=== Deploy without permissions
>>> [CLI] bundle deploy
Uploading immutable bundle snapshot...
Created internal_immutable_snapshots.immutable
Created jobs.my_job
Files: 0 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
First snapshot path: /Workspace/Users/[UUID]/.snapshots/[UUID]/[SNAPSHOT_HASH]/files/src/main.py
Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
First snapshot path: /Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/main.py

=== Add permissions and redeploy
>>> [CLI] bundle deploy
Expand All @@ -21,12 +21,12 @@ Consider using a adding a top-level permissions section such as the following:
See https://docs.databricks.com/dev-tools/bundles/permissions.html to learn more about permission configuration.
in databricks.yml:22:3

Uploading immutable bundle snapshot...
Recreated internal_immutable_snapshots.immutable
Updated jobs.my_job
Created jobs.my_job.permissions
Files: 0 uploaded, 0 deleted
Resources: 1 created, 1 changed, 0 deleted, 0 unchanged
Second snapshot path: /Workspace/Users/[UUID]/.snapshots/[UUID]/[SNAPSHOT_HASH]/files/src/main.py
Resources: 2 created, 1 changed, 1 deleted, 0 unchanged
Second snapshot path: /Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/main.py

=== Verify snapshot path changed after permissions change
OK: snapshot path changed as expected
Expand Down
139 changes: 129 additions & 10 deletions acceptance/bundle/deploy/immutable/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@ Validation OK!

>>> [CLI] bundle plan -o json
Building python_artifact...

>>> jq .plan["resources.jobs.my_job"].new_state.value.tasks plan.json
[
{
"notebook_task": {
"notebook_path": "${workspace.snapshot_path}/files/src/notebook"
"notebook_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/notebook"
},
"task_key": "notebook_task"
},
Expand All @@ -28,30 +30,147 @@ Building python_artifact...
{
"environment_key": "env",
"spark_python_task": {
"python_file": "${workspace.snapshot_path}/files/src/main.py"
"python_file": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/main.py"
},
"task_key": "spark_python_task"
}
]

>>> jq .plan["resources.jobs.my_job"].new_state.value.environments plan.json
[
{
"environment_key": "env",
"spec": {
"dependencies": [
"/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/artifacts/.internal/immutable-0.0.1-py3-none-any.whl"
],
"environment_version": "4"
}
}
]

>>> jq .plan["resources.internal_immutable_snapshots.immutable"] plan.json
{
"action": "create",
"new_state": {
"value": {
"relative_path": "[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"full_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"bundle_id": "[SNAPSHOT_HASH]",
"acl": [
{
"user_name": "[USERNAME]",
"permission_level": "CAN_READ"
}
],
"zip_path": "[TEST_TMP_DIR]/.databricks/bundle/default/snapshots/[SNAPSHOT_HASH].zip"
}
}
}

>>> [CLI] bundle deploy
Building python_artifact...
Uploading immutable bundle snapshot...
Created internal_immutable_snapshots.immutable
Created jobs.my_job
Files: 0 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
Resources: 2 created, 0 changed, 0 deleted, 0 unchanged

>>> [CLI] jobs get [NUMID]
"/Workspace/Users/[UUID]/.snapshots/[UUID]/[SNAPSHOT_HASH]/files/src/main.py"
>>> jq .settings.tasks job.json
"/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/main.py"

>>> [CLI] jobs get [NUMID]
"/Workspace/Users/[UUID]/.snapshots/[UUID]/[SNAPSHOT_HASH]/files/src/notebook"
>>> jq .settings.tasks job.json
"/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/files/src/notebook"

>>> [CLI] jobs get [NUMID]
>>> jq .settings.environments[0].spec.dependencies job.json
[
"/Workspace/Users/[UUID]/.snapshots/[UUID]/[SNAPSHOT_HASH]/artifacts/.internal/immutable-0.0.1-py3-none-any.whl"
"/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]/artifacts/.internal/immutable-0.0.1-py3-none-any.whl"
]

>>> title Check snapshot plan after deploy

=== Check snapshot plan after deploy
>>> [CLI] bundle plan -o json
Building python_artifact...

>>> jq .plan["resources.internal_immutable_snapshots.immutable"] plan.json
{
"action": "recreate",
"new_state": {
"value": {
"relative_path": "[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"full_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"bundle_id": "[SNAPSHOT_HASH]",
"acl": [
{
"user_name": "[USERNAME]",
"permission_level": "CAN_READ"
}
],
"zip_path": "[TEST_TMP_DIR]/.databricks/bundle/default/snapshots/[SNAPSHOT_HASH].zip"
}
},
"remote_state": {
"relative_path": "[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"full_path": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]"
},
"changes": {
"acl": {
"action": "skip",
"reason": "missing_in_remote",
"old": [
{
"user_name": "[USERNAME]",
"permission_level": "CAN_READ"
}
],
"new": [
{
"user_name": "[USERNAME]",
"permission_level": "CAN_READ"
}
],
"remote": null
},
"bundle_id": {
"action": "skip",
"reason": "missing_in_remote",
"old": "[SNAPSHOT_HASH]",
"new": "[SNAPSHOT_HASH]",
"remote": ""
},
"full_path": {
"action": "recreate",
"reason": "immutable",
"old": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"new": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"remote": "/Workspace/Users/[UUID]/.snapshots/[SNAPSHOT_HASH]/[SNAPSHOT_HASH]"
},
"relative_path": {
"action": "recreate",
"reason": "immutable",
"old": "[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"new": "[SNAPSHOT_HASH]/[SNAPSHOT_HASH]",
"remote": "[SNAPSHOT_HASH]/[SNAPSHOT_HASH]"
},
"zip_path": {
"action": "recreate",
"reason": "immutable",
"old": "[TEST_TMP_DIR]/.databricks/bundle/default/snapshots/[SNAPSHOT_HASH].zip",
"new": "[TEST_TMP_DIR]/.databricks/bundle/default/snapshots/[SNAPSHOT_HASH].zip",
"remote": ""
}
}
}

>>> [CLI] bundle deploy
Building python_artifact...
Recreated internal_immutable_snapshots.immutable
Updated jobs.my_job
Files: 0 uploaded, 0 deleted
Resources: 1 created, 1 changed, 1 deleted, 0 unchanged

=== Check that job uses a new snapshot: Snapshot changed as expected

>>> [CLI] bundle destroy --auto-approve
The following resources will be deleted:
delete resources.jobs.my_job
Expand Down
Loading
Loading