Skip to content

ssh: skip filesystem registration on serverless - #6856

Merged
anton-107 merged 1 commit into
mainfrom
anton-107/ssh-skip-fuse-serverless
Sep 29, 2026
Merged

anton-107 merged 1 commit into
mainfrom
anton-107/ssh-skip-fuse-serverless

Conversation

@anton-107

@anton-107 anton-107 commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Changes

On serverless, the SSH tunnel server no longer tries to register itself with the WSFS and UC-FUSE daemons. It logs one Info line at startup and skips the self-lookup, the current-user lookup, both daemon PUTs, and the one-minute refresh loop:

Skipping SSH filesystem registration on serverless; /Workspace and /Volumes access depends on the bootstrap notebook
  • New helper startFuseRegistration(ctx, client, serverless) in internal/server/fuse.go. The existing registration block and its warning moved out of Run without changes. Run now calls the helper with opts.Serverless.
  • The ssh server --serverless help text now says "Whether the server runs on serverless compute".
  • README ("Filesystem access") and FAILURE_MODES.md ("Filesystem access after the bootstrap notebook exits") now describe the serverless behaviour.
  • Dedicated clusters behave exactly as before. The fuse package is unchanged.

How Run changes at startup:

 server.Run
-  registerFuseCredentials            # always ran, warned on failure
+  startFuseRegistration(opts.Serverless)
+    if serverless
+      log.Info "Skipping SSH filesystem registration on serverless; ..."
+      return
+    registerFuseCredentials          # dedicated only, unchanged
+      fuse.Self                      # read own PID + start time from /proc
+      fuseUserInfo                   # CurrentUser.Me, 2s timeout
+      fuse.KeepRegistered            # PUT to WSFS :1021 and UC-FUSE :1015, refresh every 60s
   findAvailablePort
   SaveWorkspaceMetadata

Where filesystem access comes from, by compute type:

flowchart LR
    subgraph Dedicated
        R1[bootstrap REPL<br/>registers itself] --> S1[tunnel server]
        S1 -- registers own PID<br/>every 60s --> D1[WSFS / UC-FUSE]
        S1 --> SH1[SSH sessions<br/>keep access after REPL exits]
    end
    subgraph Serverless
        R2[bootstrap REPL<br/>registers as root] --> D2[WSFS / UC-FUSE]
        R2 --> S2[tunnel server<br/>skips registration]
        S2 --> SH2[SSH sessions<br/>covered as REPL descendants]
        S2 -. non-root UID refused<br/>SDR-3124 .-x D2
    end
Loading

Why

Skipping costs nothing. The bootstrap REPL's own registration runs as root and already covers the tunnel server and its descendants, and the container is torn down 35–60s after the REPL exits. The skip depends on the compute type the client already passes to the server, not on the error, so re-enabling registration later is one deliberate change. The regression came from #6645.

No changelog fragment: the #6645 entry ships as written.

Tests

  • New unit test TestStartFuseRegistrationSkipsServerless. Any HTTP request fails it, which proves the current-user lookup is skipped. It checks that the skip line is logged and nothing at WARN or above. It touches neither /proc nor the network, so it runs on Linux, macOS and Windows.
  • go test ./experimental/ssh/... passes, except TestSSHServerBootstrap, which fails locally on Python 3.6. That failure existed before this change.
  • ./task lint-q and ./task ws are clean.
  • Not yet run on real compute: the dogfood checks from the spec (serverless CPU, serverless GPU_1xA10, dedicated control).

This pull request and its description were written by Isaac.

On serverless the tunnel server runs as a non-root user, and the CRI shim
rejects non-root UIDs on the WSFS and UC-FUSE daemon ports. Every
registration attempt and refresh tick failed, logging ~1,440 warnings per
24h and flooding the warn-only log buffer that `ssh connect` reads back.

Gate registration on the existing --serverless option: on serverless, log
one Info line and skip the self-lookup, current-user lookup, daemon PUTs
and refresh loop. The bootstrap REPL's own registration already covers
the server's descendants. Dedicated clusters are unchanged.

Refs DECO-28664.

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 981e94d

Run: 36391631338

Env ✅​pass 🙈​skip Time
✅​ aws linux 276 15 5:35
✅​ aws windows 278 13 4:50
✅​ azure linux 275 15 6:12
✅​ azure windows 277 13 4:45
✅​ gcp linux 276 15 5:22
✅​ gcp windows 278 13 4:37
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:48 aws windows TestAccept
4:43 azure windows TestAccept
4:35 gcp windows TestAccept
4:05 aws linux TestAccept
4:04 gcp linux TestAccept
3:58 azure linux TestAccept

@anton-107
anton-107 added this pull request to the merge queue Sep 29, 2026
Merged via the queue into main with commit 7861cda Sep 29, 2026
24 checks passed
@anton-107
anton-107 deleted the anton-107/ssh-skip-fuse-serverless branch September 29, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants