Revert PR #6558: fix SSH transfer regression in v1.16.0 by anton-107 · Pull Request #6608 · databricks/cli

anton-107 · 2026-09-10T15:13:45Z

Summary

Reverts commit 3cee3a1 to fix CLI v1.16.0 regression: any continuous transfer larger than ~1 MiB over the SSH tunnel kills the session.

Impact:

databricks ssh connect --ide hits it 100% of the time (VS Code server pushes several MB during startup, session drops after 15-30s with reconnect loops)
Any SSH tunnel transfer larger than 1 MiB in either direction fails
v1.15.0 moved 7 MB fine; regression is specific to v1.16.0

Root Cause:
The replay buffer limit (proxyResumeBufferLimit = 1<<20 in experimental/ssh/internal/proxy/proxy.go) misreads ordinary in-flight data as a dead peer. When sendBuffer.append exhausts the unacknowledged window, it returns errSendWindowExhausted and kills the sending loop, even though the peer is actively sending acks. Instrumentation showed the client sends acks and every write succeeds, but the peer never applies them during continuous bursts — the window is reached by ordinary in-flight data and misinterpreted as connection death.

… of ending the session" This reverts commit 3cee3a1. Fixes v1.16.0 regression where continuous SSH transfers larger than ~1 MiB kill the session. The replay buffer limit (proxyResumeBufferLimit = 1<<20) in proxy.go misreads in-flight data as a dead peer and exhausts the send window, ending the session prematurely. Affected: databricks ssh connect --ide (100% repro after 15-30s when VS Code server transfers several MB during startup), and any SSH tunnel transfer larger than 1 MiB in either direction. The reverted code did bundle two independent fixes (pre-swap handover resilience and SSH telemetry categories) alongside the window limit. A targeted fix-forward (degrade full window to non-resumable instead of killing session) is being prepared in parallel as the preferred path. Co-authored-by: Isaac <no-reply@databricks.com>

eng-dev-ecosystem-bot · 2026-09-10T15:35:06Z

Integration test report

Commit: 673b11c

Run: 34518777883

	Env	💚RECOVERED	✅pass	🙈skip	Time
💚	aws linux	1	275	15	5:03
💚	aws windows	1	277	13	4:27
💚	azure linux	1	274	15	5:18
💚	azure windows	1	276	13	4:23
💚	gcp linux	1	275	15	5:09
💚	gcp windows	1	277	13	4:18

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
4:24	aws windows	TestAccept
4:21	azure windows	TestAccept
4:16	gcp windows	TestAccept
2:22	aws linux	TestAccept
2:18	azure linux	TestAccept
2:10	gcp linux	TestAccept

….16.0 (#6612) ## Changes Cherry-picks #6608 onto `release/v1.16.x` for the v1.16.1 patch release: - Revert of #6558 (`experimental/ssh: survive a dropped tunnel connection instead of ending the session`), reverting commit 3cee3a1, which is contained in v1.16.0. - The matching changelog fragment under `.nextchanges/cli/`, so v1.16.1's rendered changelog carries the entry. The revert applied to this branch with no conflicts and produces a patch identical to the one on `main`. ## Why v1.16.0 kills the SSH session on any continuous transfer larger than ~1 MiB. The replay buffer limit (`proxyResumeBufferLimit = 1<<20`) in `proxy.go` misreads in-flight data as a dead peer and exhausts the send window, ending the session prematurely. Affected: `databricks ssh connect --ide` (100% repro after 15-30s, when the VS Code server transfers several MB during startup), and any SSH tunnel transfer larger than 1 MiB in either direction. The reverted change bundled two independent fixes (pre-swap handover resilience and SSH telemetry categories) alongside the window limit. A targeted fix-forward on `main` (degrade a full window to non-resumable instead of killing the session) is the preferred long-term path; this patch-release branch takes the revert. ## Tests - `go build ./...` - `go test ./experimental/ssh/... ./libs/telemetry/...` — pass - `go test ./acceptance -run 'TestAccept/ssh'` — pass - `tools/validate_nextchanges.py` — pass This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

## Release v1.16.1 ### CLI * Revert tunnel resume layer to fix SSH transfer regression in v1.16.0 affecting transfers larger than 1 MiB. ([#6608](#6608), [#6612](#6612))

Already part of patch release notes

Conflict in experimental/ssh/internal/proxy/keepalive_test.go: main's #6598 restructured keepaliveTestDialer so the pausable socket is handed to the test only after the websocket handshake completes, while this branch's revert of #6558 restores the pre-resume createWebsocketConnectionFunc signature (connID string instead of DialRequest). Kept both: #6598's dialer restructure and async keystroke write, expressed against the reverted signature. Co-authored-by: Isaac <no-reply@databricks.com>

eng-dev-ecosystem-bot · 2026-09-11T00:08:29Z

Integration test report

Commit: aaab7ff

Run: 34522945619

	Env	❌FAIL	🟨KNOWN	🙈SKIP	✅pass	🙈skip	Time
❌	aws linux	481	5	1	1109	1031	200:12
❌	aws windows	435	5	1	1061	1054	238:00
❌	azure linux	393	1	3	1014	1078	163:33
❌	azure windows	348	1	3	965	1101	182:14
❌	gcp linux	383	1	3	1010	1082	176:15
❌	gcp windows	337	1	3	962	1105	190:39

494 interesting tests: 488 FAIL, 5 KNOWN, 1 SKIP

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
🟨	TestAccept	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
❌	TestAccept/bundle/apps/compute_size	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/apps/compute_size/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/apps/job_permissions	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/apps/job_permissions/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/config-remote-sync/cli_defaults	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/cli_defaults/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/config_edits	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/config_edits/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/dashboard_etag	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/dashboard_etag/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/flushed_cache	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/flushed_cache/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/formatting_preserved	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/formatting_preserved/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/job_fields	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/job_fields/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/job_multiple_tasks	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/job_multiple_tasks/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/job_params_variables	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/job_params_variables/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/job_pipeline_task	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/job_pipeline_task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/json_status_selectors	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/json_status_selectors/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/multiple_files	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/multiple_files/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/multiple_resources	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/output_json	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/output_json/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/output_no_changes	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/output_no_changes/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/pipeline_fields	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/pipeline_fields/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/resolve_variables	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/resolve_variables/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/select_basic	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/select_basic/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/select_multiple	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/select_multiple/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/skip_deployment	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/skip_deployment/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/skip_permissions	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/skip_permissions/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/target_override	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/target_override/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/config-remote-sync/task_rename_revert	❌F	🙈s	❌F	🙈s	❌F	🙈s
❌	TestAccept/bundle/config-remote-sync/task_rename_revert/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F		❌F		❌F
❌	TestAccept/bundle/deploy/empty-bundle	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/empty-bundle/DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC=true/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/files/no-snapshot-sync	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/mlops-stacks	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/spark-jar-task	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/jobs-and-pipeline	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/jobs-and-pipeline/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/root-path-name-target	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/root-path-name-target/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/root-path-not-scoped	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/root-path-not-scoped/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/sibling-target	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/destroy/sibling-target/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/declined-deploy	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/declined-deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/depends-on	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/depends-on/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/deployment-metadata-change	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/existing-state	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/existing-state/DATABRICKS_BUNDLE_ENGINE=direct/DMS=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/multiple-resources	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/multiple-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/no-drift	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/no-drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/no-resources	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/no-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/provenance	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/provenance/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/record-failure	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/record-failure/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/stale-plan	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/successful-recreate	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/generate/pipeline_and_deploy	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/generate/pipeline_and_deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/generate/python_job_and_deploy	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/generate/python_job_and_deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=app.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=app.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=catalog.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=catalog.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster_policy.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=cluster_policy.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=dashboard.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=dashboard.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=experiment.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=experiment.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=genie_space.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=instance_pool.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=instance_pool.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_for_each_task.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_for_each_task.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_job_cluster.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_job_cluster.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_task_cluster.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_apply_policy_default_values_task_cluster.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_cross_resource_ref.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_cross_resource_ref.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_escaped_refs.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_escaped_refs.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_permission_ref.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_permission_ref.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_pydabs_10_tasks.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_pydabs_10_tasks.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_run.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_run.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_with_depends_on.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_with_task.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=job_with_task.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model_serving_endpoint.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model_serving_endpoint.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model_with_permissions.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=model_with_permissions.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_allow_duplicate_names.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_apply_policy_default_values.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_apply_policy_default_values.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_config_dots.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=pipeline_config_dots.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=registered_model.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=registered_model.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema_grant_ref.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema_grant_ref.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema_uppercase_name.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=schema_uppercase_name.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret_scope.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret_scope.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret_scope_default_backend_type.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=secret_scope_default_backend_type.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=sql_warehouse.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=sql_warehouse.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=vector_search_endpoint.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=vector_search_endpoint.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=vector_search_index.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=vector_search_index.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume_path_job_ref.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume_path_job_ref.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume_uppercase_name.yml.tmpl/READPLAN=	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/invariant/no_drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/INPUT_CONFIG=volume_uppercase_name.yml.tmpl/READPLAN=1	❌F	❌F	❌F	❌F	❌F	❌F
❌	TestAccept/bundle/local_state_staleness	❌F	❌F	❌F	❌F	❌F	❌F

Top 50 slowest tests (at least 2 minutes):

duration	env	testname
27:46	aws windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
24:37	aws windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
14:54	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
13:39	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
12:37	aws windows	TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
11:56	aws linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
11:50	aws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
8:52	aws linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:50	gcp windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:47	aws windows	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:28	gcp linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
8:28	azure windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:52	aws windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:50	azure windows	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:35	gcp linux	TestAccept/bundle/resources/clusters/deploy/simple/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:29	azure linux	TestAccept/bundle/resources/apps/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:07	gcp linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
7:04	gcp linux	TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:56	aws linux	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:50	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:50	gcp windows	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:45	azure windows	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:35	gcp windows	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
6:25	aws linux	TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:23	aws linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:56	azure linux	TestAccept/bundle/resources/clusters/lifecycle-started/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:32	aws linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
5:28	gcp windows	TestAccept/bundle/resources/registered_models/basic/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
5:24	aws linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:18	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:17	aws windows	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
5:16	gcp linux	TestAccept/bundle/resources/apps/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:15	azure linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
5:13	aws linux	TestAccept/bundle/invariant/destroy_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=
5:12	gcp linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
5:07	gcp linux	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:03	azure linux	TestAccept/bundle/config-remote-sync/multiple_resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
5:01	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:59	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:49	aws windows	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:48	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:43	gcp linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:36	gcp linux	TestAccept/bundle/deploy/spark-jar-task/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:27	aws linux	TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster.yml.tmpl/READPLAN=1
4:17	aws linux	TestAccept/bundle/resources/apps/lifecycle-started-omitted/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:14	aws linux	TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=
4:09	aws windows	TestAccept/bundle/invariant/delete_idempotent/DATABRICKS_BUNDLE_ENGINE=direct/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
4:06	azure linux	TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:05	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct/DMS=
4:02	azure windows	TestAccept/bundle/quiet-levels/DATABRICKS_BUNDLE_ENGINE=terraform/DMS=

(316 table rows omitted to keep the report under 60000 bytes)

#6558 capped the tunnel's unacknowledged replay window at 1 MiB and read a full window as a peer that had stopped acknowledging, so any transfer past that ended the session. It shipped in v1.16.0, broke `ssh connect --ide` outright, and was reverted in #6608. No test noticed, in pre- or post-merge CI. This test moves 8 MiB in each direction and asserts the byte counts. It is the first ssh test to run on cloud since #4838 disabled the old cloud-only ones as too flaky; serverless CPU has no cluster to provision and no accelerator to wait for, and the cloud run costs about 25s, most of it serverless startup rather than the transfer. The byte counts are the assertion rather than the exit code, because the session could end with truncated output and exit code 0. Locally the test server never negotiates the resume protocol, so the local run covers the plumbing over a real ssh session and the cloud run is what would catch the window coming back. Co-authored-by: Isaac <no-reply@databricks.com>

…atabricks#6626) ## Why databricks#6558 capped the tunnel's unacknowledged replay window at 1 MiB and read a full window as a peer that had stopped acknowledging, so ordinary in-flight data ended the session: ``` Proxy server error: proxy websocket dropped failed to send message: resume send buffer is full: the peer stopped acknowledging: 1045563 unacknowledged bytes, limit 1048576 ``` It shipped in v1.16.0, broke `ssh connect --ide` outright (the remote IDE server pushes several megabytes as it starts), and was reverted in databricks#6608. **No test noticed, in pre- or post-merge CI** — this PR is that missing coverage, so a re-introduction is caught before users are. ## What it does `acceptance/ssh/bulk-transfer` moves 8 MiB in each direction over a real `ssh connect` session and asserts the byte counts. Measured against a real workspace, on the tree as it was before the revert and after: | | result | |---|---| | local (test server's sshd over `ws://`) | pass, 3.4s | | cloud, serverless CPU, with databricks#6558 present | **fail** — `received 1032192 of 4194304`, `delivered 0 of 4194304` | | cloud, serverless CPU, with databricks#6558 out of the path | pass, 23s cold start | | cloud, on this PR's own CI run | pass, 72s (cold serverless start plus the one-time binary upload) | Some details worth flagging for review: - **8 MiB, and it must stay under 10,000,000.** The root `test.toml` rewrites any run of 8+ digits to `[NUMID]`, which would silently erase the counts and leave the test asserting nothing. 8 MiB is also the size that matters — roughly what the IDE server pushes at startup. - **The byte counts are the assertion, not the exit code.** The regression could end the session with truncated output *and* exit code 0, so an exit-code check would have passed on it. - **First ssh test to run on cloud since databricks#4838** disabled the old cloud-only ones as too flaky. Serverless CPU has no cluster to provision and no accelerator to wait for, which is what makes this one dependable. - **Deliberately not `CloudSlow`.** PR cloud runs pass `-short` (`task cloud-select`), so `CloudSlow` would keep this off every PR — including one that changes the test itself — and leave it only to the full suite on main. At ~70s in CI - a cold serverless start plus the one-time upload of the two release archives, not the transfer - it does not belong with the multi-minute infra tests that carry that flag. - **Locally the test server never negotiates the resume protocol**, so the local run covers the plumbing over a real ssh session (real sshd, real handshake, 16 MiB through the websocket) and the cloud run is what would catch the window coming back. ## Tests - `go test ./acceptance -run TestAccept/ssh` — pass - `go test ./experimental/ssh/...` — pass - cloud runs on serverless CPU in both configurations (table above) - this PR's own `Integration Tests` check ran it and nothing else: the cloud leg's selection resolves to `^ssh$/^bulk-transfer$`, and its log shows `PASS acceptance.TestAccept/ssh/bulk-transfer (72.08s)` against a real serverless tunnel This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>

## Summary Reverts commit 3cee3a1 to fix CLI v1.16.0 regression: any continuous transfer larger than ~1 MiB over the SSH tunnel kills the session. **Impact:** - `databricks ssh connect --ide` hits it 100% of the time (VS Code server pushes several MB during startup, session drops after 15-30s with reconnect loops) - Any SSH tunnel transfer larger than 1 MiB in either direction fails - v1.15.0 moved 7 MB fine; regression is specific to v1.16.0 **Root Cause:** The replay buffer limit (`proxyResumeBufferLimit = 1<<20` in `experimental/ssh/internal/proxy/proxy.go`) misreads ordinary in-flight data as a dead peer. When `sendBuffer.append` exhausts the unacknowledged window, it returns `errSendWindowExhausted` and kills the sending loop, even though the peer is actively sending acks. Instrumentation showed the client sends acks and every write succeeds, but the peer never applies them during continuous bursts — the window is reached by ordinary in-flight data and misinterpreted as connection death. --------- Co-authored-by: Isaac <no-reply@databricks.com> Co-authored-by: Jan N Rose <janniklas.rose@gmail.com>

…6626) ## Why #6558 capped the tunnel's unacknowledged replay window at 1 MiB and read a full window as a peer that had stopped acknowledging, so ordinary in-flight data ended the session: ``` Proxy server error: proxy websocket dropped failed to send message: resume send buffer is full: the peer stopped acknowledging: 1045563 unacknowledged bytes, limit 1048576 ``` It shipped in v1.16.0, broke `ssh connect --ide` outright (the remote IDE server pushes several megabytes as it starts), and was reverted in #6608. **No test noticed, in pre- or post-merge CI** — this PR is that missing coverage, so a re-introduction is caught before users are. ## What it does `acceptance/ssh/bulk-transfer` moves 8 MiB in each direction over a real `ssh connect` session and asserts the byte counts. Measured against a real workspace, on the tree as it was before the revert and after: | | result | |---|---| | local (test server's sshd over `ws://`) | pass, 3.4s | | cloud, serverless CPU, with #6558 present | **fail** — `received 1032192 of 4194304`, `delivered 0 of 4194304` | | cloud, serverless CPU, with #6558 out of the path | pass, 23s cold start | | cloud, on this PR's own CI run | pass, 72s (cold serverless start plus the one-time binary upload) | Some details worth flagging for review: - **8 MiB, and it must stay under 10,000,000.** The root `test.toml` rewrites any run of 8+ digits to `[NUMID]`, which would silently erase the counts and leave the test asserting nothing. 8 MiB is also the size that matters — roughly what the IDE server pushes at startup. - **The byte counts are the assertion, not the exit code.** The regression could end the session with truncated output *and* exit code 0, so an exit-code check would have passed on it. - **First ssh test to run on cloud since #4838** disabled the old cloud-only ones as too flaky. Serverless CPU has no cluster to provision and no accelerator to wait for, which is what makes this one dependable. - **Deliberately not `CloudSlow`.** PR cloud runs pass `-short` (`task cloud-select`), so `CloudSlow` would keep this off every PR — including one that changes the test itself — and leave it only to the full suite on main. At ~70s in CI - a cold serverless start plus the one-time upload of the two release archives, not the transfer - it does not belong with the multi-minute infra tests that carry that flag. - **Locally the test server never negotiates the resume protocol**, so the local run covers the plumbing over a real ssh session (real sshd, real handshake, 16 MiB through the websocket) and the cloud run is what would catch the window coming back. ## Tests - `go test ./acceptance -run TestAccept/ssh` — pass - `go test ./experimental/ssh/...` — pass - cloud runs on serverless CPU in both configurations (table above) - this PR's own `Integration Tests` check ran it and nothing else: the cloud leg's selection resolves to `^ssh$/^bulk-transfer$`, and its log shows `PASS acceptance.TestAccept/ssh/bulk-transfer (72.08s)` against a real serverless tunnel This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>

anton-107 and others added 3 commits September 10, 2026 17:03

Add changelog fragment for PR #6558 revert

764d9d1

Update changelog fragment with correct PR number 6608

6a952b2

anton-107 requested review from janniklasrose and rugpanov September 10, 2026 15:15

anton-107 marked this pull request as ready for review September 10, 2026 15:15

janniklasrose approved these changes Sep 10, 2026

View reviewed changes

anton-107 marked this pull request as draft September 10, 2026 15:20

anton-107 marked this pull request as ready for review September 10, 2026 15:25

anton-107 mentioned this pull request Sep 10, 2026

Revert PR #6558 on release/v1.16.x: fix SSH transfer regression in v1.16.0 #6612

Merged

janniklasrose approved these changes Sep 10, 2026

View reviewed changes

deco-sdk-tagging Bot added a commit that referenced this pull request Sep 10, 2026

[Release] Release v1.16.1

e7f04d0

## Release v1.16.1 ### CLI * Revert tunnel resume layer to fix SSH transfer regression in v1.16.0 affecting transfers larger than 1 MiB. ([#6608](#6608), [#6612](#6612))

Delete .nextchanges/cli/revert-6558-tunnel-regression.md

3700430

Already part of patch release notes

janniklasrose enabled auto-merge September 10, 2026 16:35

anton-107 and others added 2 commits September 10, 2026 20:50

Merge origin/main into revert-6558-ssh-tunnel-regression

673b11c

janniklasrose added this pull request to the merge queue Sep 10, 2026

Merged via the queue into main with commit aaab7ff Sep 10, 2026
34 checks passed

janniklasrose deleted the revert-6558-ssh-tunnel-regression branch September 10, 2026 19:51

anton-107 mentioned this pull request Sep 11, 2026

acceptance: cover ssh tunnel transfers larger than the resume window #6626

Merged

Navigation Menu

Repository navigation

Revert PR #6558: fix SSH transfer regression in v1.16.0 - #6608

Revert PR #6558: fix SSH transfer regression in v1.16.0#6608
janniklasrose merged 6 commits into
mainfrom
revert-6558-ssh-tunnel-regression

anton-107 commented Sep 10, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

anton-107 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

eng-dev-ecosystem-bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented Sep 11, 2026

Integration test report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anton-107 commented Sep 10, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Sep 10, 2026 •

edited

Loading