Write initial snapshot memory in place by ludfjig · Pull Request #1927 · hyperlight-dev/hyperlight

ludfjig · 2026-10-12T20:01:02Z

New sandboxes skip copying their initial memory. Snapshot::from_env writes the guest code, PEB, init data and page tables in place, and the host populates the rest, mostly the heap, on first guest touch.

Benchmarks

Compared to main. MSHV runs Linux 6.6.137.mshv2-5 and 6.6.100.mshv1-2. KVM runs Linux 6.17.

Benchmark	MSHV 6.6.137	MSHV 6.6.100	KVM
`sandboxes/create_initialized/small`	4.6 ms → 3.09 ms (−33%)	4.22 ms → 2.89 ms (−32%)	5.12 ms → 2.92 ms (−43%)
`sandboxes/create_initialized/medium`	11.8 ms → 5.93 ms (−50%)	11.6 ms → 9.15 ms (−21%)	9.81 ms → 3.11 ms (−68%)
`sandboxes/create_initialized/large`	38.5 ms → 11 ms (−71%)	39.3 ms → 29.2 ms (−26%)	28.8 ms → 3.59 ms (−88%)
`snapshot_files/cold_start_via_evolve/large`	38.8 ms → 10.7 ms (−72%)	40.7 ms → 28.6 ms (−30%)	29.1 ms → 4.24 ms (−85%)

6.6.100 gains less because it pins guest memory, while 6.6.137 maps it as movable regions. On 6.6.137, a fresh sandbox's first call that writes 32 MiB of heap is 4% slower. Other benchmarks are unchanged.

Copilot

🟡 Changes recommended

Lazy no-reserve allocations need aggregate admission or commitment accounting to prevent deferred host-wide OOM.

1 open finding

Add memory admission control for lazy large-heap allocations · New

What changed in this PR

Optimizes sandbox creation by constructing initial snapshot memory directly in shared memory and relying on lazy page population.

Changes:

Writes guest initialization data in place.
Freezes shared memory without copying.
Documents the performance improvement.

File	Description
`src/hyperlight_host/src/sandbox/snapshot/mod.rs`	Builds snapshots directly in shared memory.
`src/hyperlight_host/src/mem/shared_mem.rs`	Restricts the copying helper to tests and HVF.
`CHANGELOG.md`	Documents lazy memory population.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Snapshot::from_env writes the guest code, PEB and init data straight into shared memory and freezes it. The host populates the remaining pages on first guest touch, so sandbox creation cost stays independent of heap size. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

hyperlight-gh-bot · 2026-10-30T20:01:05Z

Benchmark Results

Measured commit: b50c34e97f80
Baseline commit: b99eb5f65293

kvm / amd (Linux) (🌟 6.06x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🌟 6.06x faster

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`373.16 µs` (➖ 1.00x slower)
`byte_chunks`		`756.70 ns` (➖ 1.01x slower)
`vec_bytes`		`585.62 ns` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`143.58 ns` (➖ 1.01x slower)
`262144`	`521.12 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.42 ms` (🌟 6.06x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.68 ns` (➖ 1.00x slower)	`7.68 ns` (➖ 1.00x slower)	`7.68 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`117.57 µs` (➖ 1.00x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`1.93 µs` (➖ 1.08x faster)	`2.05 µs` (➖ 1.07x slower)
`262144`	`7.45 µs` (➖ 1.03x slower)	`7.22 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.02 µs` (➖ 1.06x faster)	`1.02 µs` (➖ 1.01x slower)
`65536`	`6.19 µs` (➖ 1.02x faster)	`6.29 µs` (➖ 1.01x faster)
`262144`	`28.33 µs` (➖ 1.06x slower)

kvm / intel (Linux) (🌟 5.41x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🌟 5.41x faster

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`726.87 µs` (➖ 1.03x faster)
`byte_chunks`		`765.42 ns` (➖ 1.00x faster)
`vec_bytes`		`577.01 ns` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`204.45 ns` (➖ 1.05x slower)
`262144`	`592.16 ns` (➖ 1.05x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`15.81 ms` (🌟 5.41x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.11 ns` (➖ 1.06x slower)	`8.10 ns` (➖ 1.05x slower)	`8.14 ns` (➖ 1.06x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`61.39 µs` (➖ 1.07x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.52 µs` (➖ 1.05x slower)	`2.51 µs` (➖ 1.04x slower)
`262144`	`8.96 µs` (➖ 1.05x slower)	`9.03 µs` (➖ 1.06x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`856.78 ns` (➖ 1.07x slower)	`857.79 ns` (➖ 1.05x slower)
`65536`	`8.35 µs` (➖ 1.03x slower)	`8.39 µs` (➖ 1.03x slower)
`262144`	`35.65 µs` (➖ 1.05x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`353.18 µs` (➖ 1.28x slower)
`byte_chunks`		`940.88 ns` (➖ 1.04x faster)
`vec_bytes`		`725.62 ns` (➖ 1.00x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`192.64 ns` (➖ 1.02x faster)
`262144`	`729.46 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`66.60 ms` (➖ 1.13x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.54 ns` (➖ 1.01x faster)	`9.52 ns` (➖ 1.03x faster)	`9.68 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`106.11 µs` (➖ 1.04x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.22 µs` (➖ 1.00x slower)	`2.42 µs` (➖ 1.00x faster)
`262144`	`9.82 µs` (➖ 1.01x slower)	`10.19 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.24 µs` (➖ 1.02x slower)	`1.24 µs` (➖ 1.03x faster)
`65536`	`8.12 µs` (➖ 1.18x faster)	`8.13 µs` (➖ 1.27x faster)
`262144`	`35.66 µs` (➖ 1.09x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`670.22 µs` (➖ 1.01x faster)
`byte_chunks`		`937.81 ns` (➖ 1.01x slower)
`vec_bytes`		`655.49 ns` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`259.76 ns` (➖ 1.07x slower)
`262144`	`660.63 ns` (➖ 1.00x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`71.28 ms` (➖ 1.17x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.68 ns` (➖ 1.00x faster)	`8.69 ns` (➖ 1.00x faster)	`8.68 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`54.10 µs` (➖ 1.00x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.29 µs` (➖ 1.00x faster)	`2.26 µs` (➖ 1.01x slower)
`262144`	`7.89 µs` (➖ 1.02x faster)	`7.98 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`895.42 ns` (➖ 1.01x faster)	`896.97 ns` (➖ 1.01x faster)
`65536`	`7.50 µs` (➖ 1.01x faster)	`7.54 µs` (➖ 1.00x slower)
`262144`	`41.61 µs` (➖ 1.02x slower)

hyperv-ws2025 / amd (Windows) (🌟 7.79x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🌟 7.79x faster

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`2.17 ms` (➖ 1.00x faster)
`byte_chunks`		`1.25 µs` (➖ 1.10x slower)
`vec_bytes`		`846.44 ns` (➖ 1.09x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`230.21 ns` (➖ 1.00x slower)
`262144`	`787.94 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.90 ms` (🌟 7.79x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.06 ns` (➖ 1.03x faster)	`10.05 ns` (➖ 1.00x slower)	`13.21 ns` (➖ 1.32x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`906.15 µs` (➖ 1.15x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.41 µs` (➖ 1.05x faster)	`2.31 µs` (➖ 1.04x faster)
`262144`	`9.13 µs` (➖ 1.03x faster)	`9.24 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.30 µs` (➖ 1.01x slower)	`1.50 µs` (➖ 1.03x slower)
`65536`	`8.86 µs` (➖ 1.10x faster)	`9.07 µs` (➖ 1.03x faster)
`262144`	`38.66 µs` (➖ 1.01x faster)

hyperv-ws2025 / intel (Windows) (🌟 7.87x)

Top improvements

sandboxes/create_initialized_and_drop/medium — 🌟 7.87x faster

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`2.87 ms` (➖ 1.09x faster)
`byte_chunks`		`1.17 µs` (➖ 1.00x faster)
`vec_bytes`		`765.20 ns` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`210.05 ns` (➖ 1.00x slower)
`262144`	`748.98 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.42 ms` (🌟 7.87x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.15 ns` (➖ 1.06x faster)	`9.71 ns` (➖ 1.26x faster)	`9.65 ns` (➖ 1.28x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`745.94 µs` (➖ 1.10x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.51 µs` (➖ 1.08x slower)	`2.30 µs` (➖ 1.02x slower)
`262144`	`7.53 µs` (➖ 1.01x faster)	`7.64 µs` (➖ 1.00x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.08 µs` (➖ 1.02x slower)	`1.39 µs` (➖ 1.05x faster)
`65536`	`9.67 µs` (➖ 1.13x slower)	`8.37 µs` (➖ 1.01x slower)
`262144`	`41.90 µs` (➖ 1.21x faster)

_{Reported by cargo ci bench-report --candidate run:41113574968 --config-file bench_report.toml.}

jsturtevant · 2026-10-30T21:29:27Z

it is interesting that benchmarks are reporting increases for everything but mshv 🤔

ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 12, 2026

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners October 12, 2026 20:01

ludfjig added this pull request to stack #1929 October 12, 2026 20:01

ludfjig mentioned this pull request Oct 12, 2026

Layered/incremental snapshots #1902

Closed

This comment has been minimized.

Sign in to view

Copilot AI balanced review requested due to automatic review settings October 13, 2026 18:56

ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 96fab64 to ab775e0 Compare October 13, 2026 18:56

Copilot started reviewing on behalf of ludfjig October 13, 2026 19:03

View session

Copilot AI reviewed Oct 13, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs Resolved

ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from ab775e0 to 107a858 Compare October 13, 2026 21:01

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 107a858 to aa55340 Compare October 13, 2026 23:00

This comment has been minimized.

Sign in to view

andreiltd previously approved these changes Oct 14, 2026

View reviewed changes

ludfjig dismissed andreiltd’s stale review via 421a453 October 15, 2026 22:00

ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from aa55340 to 421a453 Compare October 15, 2026 22:00

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 421a453 to 5631afe Compare October 21, 2026 19:27

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 5631afe to 5c04ce3 Compare October 26, 2026 18:44

Base automatically changed from ll/1-layered-snapshots to main October 30, 2026 18:45

ludfjig force-pushed the ll/2-lazy-sandbox-memory branch from 5c04ce3 to b50c34e Compare October 30, 2026 19:14

jsturtevant approved these changes Oct 30, 2026

View reviewed changes

ludfjig merged commit d51a3f3 into main Oct 30, 2026
106 of 108 checks passed

ludfjig deleted the ll/2-lazy-sandbox-memory branch October 30, 2026 22:13

Conversation

ludfjig commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Uh oh!

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Oct 30, 2026

Benchmark Results

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top improvements

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

jsturtevant commented Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

ludfjig commented Oct 12, 2026 •

edited

Loading