Repository navigation
direct: store genie space serialized_space in state as a content hash - #6707
Merged
Merged
Conversation
Mirror the serialized_dashboard hashing: declare serialized_space in hashed_fields so state persists only its sha256 content hash instead of the full inlined JSON. The full content still reaches the API and stays in the plan's new_state; serialized_space remains ignore_remote_changes (etag_based). Adds unit tests (compaction + state-rules) and an acceptance test mirroring change-serialized-dashboard. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/genie-spaces-hashing
branch
from
September 16, 2026 11:57
3569568 to
7b71af6
Compare
denik
reviewed
Sep 16, 2026
…pace The inline test's out.plan.json still had full serialized_space content in the changes diff; regenerate it to reflect the content-hash storage. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/genie-spaces-hashing
branch
from
September 16, 2026 12:17
b42516b to
2601e88
Compare
denik
reviewed
Sep 16, 2026
Sankalp-Mittal
force-pushed
the
sankalp-mittal/genie-spaces-hashing
branch
from
September 16, 2026 12:27
2601e88 to
bc033d4
Compare
The state-rules unit test only re-asserted the declarative hashed_fields / ignore_remote_changes config; that behavior is already covered by the change-serialized-space acceptance test. Also tidies a few test comments. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/genie-spaces-hashing
branch
from
September 16, 2026 12:31
bc033d4 to
00bd412
Compare
On Windows, update_file.py rewrites the fixture in text mode, so the edited serialized_space payload ends with \r\n and diverged from the LF golden output. Add a Repls entry to normalize it, matching the dashboards change-serialized test's approach. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/genie-spaces-hashing
branch
from
September 16, 2026 13:07
5f362db to
5a547a5
Compare
denik
reviewed
Sep 16, 2026
| "databricks.yml", | ||
| ] | ||
|
|
||
| # update_file.py rewrites the fixture in text mode, so on Windows the edited |
Contributor
There was a problem hiding this comment.
we should add option to update_file.py --binary or just make it binary for all
Contributor
Author
There was a problem hiding this comment.
makes sense to make it binary for all, I'll do it in a follow up
denik
approved these changes
Sep 16, 2026
Sankalp-Mittal
enabled auto-merge
September 16, 2026 13:15
Collaborator
Integration test reportCommit: 1fbd5f7
Top 11 slowest tests (at least 2 minutes):
|
The real Genie API rejects the fixture's text_instruction.id 'ti-001'
("Expected lowercase 32-hex UUID without hyphens"), so the Cloud=true run
failed in every workspace while the local testserver accepted it. Use a valid
32-hex id (no long digit run so [NUMID] doesn't rewrite it in the golden).
On cloud the SDK retries the create, recording a duplicate consecutive POST;
collapse it with print_requests.py --unique so the request assertion is stable.
Verified against a real workspace via deco (aws-cli).
Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 3aa4ded
124 interesting tests: 56 FAIL, 37 MISS, 31 flaky
Top 50 slowest tests (at least 2 minutes):
|
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Sep 24, 2026
## Release v1.18.0 ### CLI * The AI Runtime commands have moved to `databricks air`. The previous `databricks experimental air` path now directs users to the new command. ([#6722](#6722)) * Write local state, cache, and config files atomically so an interrupted or concurrent write cannot corrupt them. ([#6708](#6708)) * Deprecate `--region` in `databricks auth docker configure` ahead of its removal in the next release, infer the Artifact Registry region when it is omitted, and add `databricks auth docker host --profile <name>` to show the profile's registry host and credential-helper status. ([#6782](#6782)) * Return `UNAUTHENTICATED` instead of `INVALID_REFRESH_TOKEN` when `databricks auth token --output json` cannot refresh a cached U2M token. ([#6731](#6731)) * Retry the current-user (SCIM `Me`) lookup on transient HTTP 500 responses so a temporarily-unavailable backend no longer fails bundle commands outright. ([#6766](#6766)) * Preserve workspace-file and volume access for SSH server descendants when the bootstrap notebook exits and the server survives. ([#6645](#6645)) * `ssh connect` and `ssh setup` now accept a `--keep-detached-processes` flag to keep processes detached from the SSH session (`tmux`, `setsid`, `nohup`) running after the tunnel shuts down. Teardown then terminates only the tunnel's own process group, and the bootstrap job run is held open while any detached process is still running, so the survivors keep their `/Workspace` and `/Volumes` access. A held-open run also suppresses cluster autotermination, so the flag is off by default, is bounded by `--server-timeout`, and is dedicated-cluster only. Without it, the server now logs a warning naming the detached processes it is about to destroy, instead of sweeping them silently. ([#6387](#6387)) ### Bundles * direct: Allow clearing a catalog's or schema's `custom_max_retention_hours` by removing it from configuration. ([#6792](#6792)) * direct: Allow clearing a genie space's `description` and a secret's `comment` by removing them from configuration. ([#6789](#6789)) * Fix direct-engine deploy recreating an MLflow experiment on every deploy when its `trace_location` was set out-of-band. ([#6787](#6787)) * direct: Store a Genie space's `serialized_space` in state as a content hash instead of its full contents. ([#6707](#6707)) * Fix `bundle deploy` failing with "Invalid python file reference" for jobs that use `git_source` with a `spark_python_task` on the direct engine. ([#6751](#6751)) * Fixed the direct engine mishandling UC grants that combine `ALL_PRIVILEGES` with a privilege it does not imply (`MANAGE`, `READ_METADATA`, `EXTERNAL_USE_SCHEMA`, `EXTERNAL_USE_LOCATION`): such privileges were dropped when granted and left behind when revoked, so the deployment never converged. ([#6733](#6733), [#6743](#6743)) * Don't fail migration if clean up actions fail. ([#6772](#6772)) * Ignore the backend-provided `spark.sql.ansi.enabled: "true"` pipeline configuration default when detecting direct-engine drift. ([#6816](#6816)) * Fix recreating a postgres synced table sometimes failing with a 409 ALREADY_EXISTS error while the previous table is still being deleted. ([#6728](#6728)) * Direct engine no longer recreates a resource when an immutable field the config omits was populated by the backend. ([#6790](#6790)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#6723](#6723)) * Bump `github.com/databricks/databricks-sdk-go` from v0.178.0 to v0.182.0. ([#6817](#6817)) * Bump the Databricks Terraform provider from v1.132.0 to v1.134.0. ([#6818](#6818))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Follow-up to #6105 (which did the same for dashboards'
serialized_dashboard). The direct-deploy engine stored a Genie space's full inlinedserialized_spaceJSON in state, where it is only ever compared for equality ("did the content change since the last deploy?") and never read back for its value. This stores it as asha256:<hex>content hash instead.serialized_spaceinhashed_fields(bundle/direct/dresources/configs/genie_spaces.yml), soCompactStatepersists only its hash. The full content still goes to the API on create/update and stays in the plan'snew_state.serialized_spaceremainsignore_remote_changes(etag-based); the two declarations are orthogonal.Tests
TestGenieSpaceCompactStateandTestGenieSpaceSerializedSpaceStateRules, mirroring the dashboard tests.TestHashedFieldsAreValidalready covers the new field generically.genie_spaces/change-serialized-spacemirrorschange-serialized-dashboard— deploy, no-op re-plan, editserialized_space→ update (not recreate), asserting the hash is stored in state and the full content reaches the create (POST) and update (PATCH) APIs. Runs locally and on a real workspace (Cloud = true).This pull request and its description were written by Isaac.