Skip to content

feat(bundle): store each distinct canister and plugin wasm once per bundle - #880

Open
raymondk wants to merge 3 commits into
mainfrom
fm/icp-cli-bundle-dedup-ship
Open

raymondk wants to merge 3 commits into
mainfrom
fm/icp-cli-bundle-dedup-ship

Conversation

@raymondk

@raymondk raymondk commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Intent

Improve bundling in dfinity/icp-cli: today, when a project is bundled, the plugin for a wasm is potentially duplicated multiple times. Instead, the bundle should have a single copy per wasm hash in the entire bundle.

A scout explored this and proposed a plan (full report: /hoststorage/dev/firstmate/data/icp-cli-bundle-dedup-plan-f2/report.md). Substance: today prepare_plugin_step in crates/icp-project/src/operations/bundle.rs writes one copy per (instance, canister, step) at <instance>/plugins/<canister>/<idx>.wasm, so identical plugins are stored repeatedly (a demo with four asset canisters held five identical 3.6 MB copies). The plan stores each distinct plugin once at the archive root as plugins/<sha256>.wasm; each step keeps its path: and sha256: fields, with a dependency instance reaching the store via a relative ../.. path. Readers (icp CLI trusted and --untrusted, imcp2-deploy, open-cloud deployer) follow path: and verify sha256:, so no reader changes. The one behaviour change: a dependency inside an extracted bundle no longer works on its own (as an untrusted root or copied out), because its plugin lives at the bundle root.

The tradeoff for the subprojects not working outside of the root is acceptable for bundles - open a new PR against icp cli with this change.

We can actually expand the plugin dedup to also dedup canister wasms. This is more efficient if canisters in the same project are using the same wasm. Look into the feasibility and if it's doable amend the PR to include that as well.

What Changed

  • icp project bundle now writes each distinct wasm only once for the whole archive. It is stored at the archive root and named by the SHA-256 of its bytes: canister wasms go to canisters/<sha256>.wasm, which replaces the per-canister <instance>/canisters/<canister>.wasm, and plugin wasms go to plugins/<sha256>.wasm, which replaces the per-step <instance>/plugins/<canister>/<idx>.wasm. In crates/icp-project/src/operations/bundle.rs, the artifact lists are now BTreeMaps keyed by store path. This removes duplicate copies and keeps the entry order deterministic.
  • Each build and sync step still names its module by path: and sha256:, so tools that read bundles need no change. A dependency instance reaches the shared store through a relative path that rises to the archive root, for example ../../canisters/<sha256>.wasm. One behaviour changes as a result: a dependency's directory in an extracted bundle no longer works on its own, whether it is copied out or used as the root of --untrusted. The bundle has to be used from its root. The CHANGELOG and docs/concepts/project-dependencies.md describe the new layout and this limitation.
  • Adds tests in bundle_tests.rs and untrusted_tests.rs for the deduplicated layout and the shared-store references. Also bumps libz-sys 1.1.31 → 1.1.33 in Cargo.lock to clear the RUSTSEC-2026-0544 advisory.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: This is a small, well-bounded layout change. Wasm modules are now stored under their own sha256 in BTreeMaps, so the dedup and the archive order are both deterministic. Every pre-built and plugin step still carries path: and sha256:, a dependency reaches the root store through a correct relative ../.. path, the existing collision check still covers the shared store directories, and the intended tradeoff (a member no longer works outside the bundle root) is documented and tested.

Testing

I built the real icp binary and downloaded the pinned network launcher into the worktree. With them I bundled a disposable four-canister project that includes a dependency. The archive holds exactly two canister wasms, one per distinct hash. The root manifest names them as canisters/&lt;sha&gt;.wasm and the dependency names its wasm as ../../canisters/&lt;sha&gt;.wasm. The extracted bundle deployed on a local PocketIC network, and all four canisters answered greet. A tampered store file was rejected by the sha256 check. A dependency copied out of the bundle failed to build because its wasm lives at the bundle root, which is the tradeoff the user accepted. The targeted bundle and untrusted integration tests passed, including deploy-from-bundle with a plugin sync step in a dependency and plugin-wasm dedup. I removed the disposable project, the local network and the build outputs. Only the evidence transcripts remain.

  • Live validation: ✅ go - 6 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Bundling a project whose canisters (root and dependency) share a wasm stores one canister wasm per sha256 at canisters/<sha256>.wasm ✅ pass live manual_bundle_transcript.txt: tar listing shows only canisters/17a05e...wasm and canisters/353ef0...wasm for 4 canisters; also bundle_stores_each_canister_wasm_once
Root manifest names the store directly and the dependency manifest reaches it via ../.., each with a matching sha256 ✅ pass live manual_bundle_transcript.txt grep of pre-built steps; bundle_rewrites_dependency_paths_to_archive_locations
Extracted bundle deploys on a local network and every canister (including the dependency's) answers greet ✅ pass live manual_deploy_transcript.txt: deploy exit=0, all 4 greet calls return ("Hello, bundle!"); bundle_and_deploy and bundle_with_dependency_deploys passed
Identical plugin wasms used by several sync steps or instances are stored once at plugins/<sha256>.wasm, and the dependency's plugin runs on deploy ✅ pass live bundle_tests.log: bundle_stores_each_plugin_wasm_once, bundle_archives_aliased_plugin_paths_once, bundle_packages_plugin_sync_steps, bundle_with_dependency_deploys (dependency plugin seeds data during…
Adversarial: a tampered store entry is rejected by the sha256 check at build ✅ pass live manual_adversarial_transcript.txt: 'checksum mismatch, expected: 353ef..., actual: 17a05...' exit=1
Accepted tradeoff: a dependency copied out of the bundle can no longer build on its own; inside the bundle it builds as untrusted ✅ pass live manual_adversarial_transcript.txt: failed to read ../../canisters/<sha>.wasm, exit=1; untrusted_tests.log: a_bundle_member_reaches_its_wasms_at_the_bundle_root ok
Evidence: Bundle listing and rewritten manifests (4 canisters, 2 distinct wasms → 2 store entries)
$ sha256sum hello.wasm vendor/lib/lib.wasm other.wasm
17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a  hello.wasm
17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a  vendor/lib/lib.wasm
353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d  other.wasm

$ icp project bundle --output bundle.tar.gz
exit=0

$ tar tzvf bundle.tar.gz
-rw-r--r-- 0/0             973 1970-01-01 00:00 icp.yaml
-rw-r--r-- 0/0             323 1970-01-01 00:00 vendor/lib/icp.yaml
-rw-r--r-- 0/0          205794 1970-01-01 00:00 canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
-rw-r--r-- 0/0          205801 1970-01-01 00:00 canisters/353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d.wasm

$ grep -A2 "pre-built" x/icp.yaml x/vendor/lib/icp.yaml
x/vendor/lib/icp.yaml:    - type: pre-built
x/vendor/lib/icp.yaml-      path: ../../canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
x/vendor/lib/icp.yaml-      sha256: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a
x/icp.yaml:    - type: pre-built
x/icp.yaml-      path: canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
x/icp.yaml-      sha256: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a
--
x/icp.yaml:    - type: pre-built
x/icp.yaml-      path: canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
x/icp.yaml-      sha256: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a
--
x/icp.yaml:    - type: pre-built
x/icp.yaml-      path: canisters/353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d.wasm
x/icp.yaml-      sha256: 353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d
Evidence: Deploy from extracted bundle on local network + greet calls
$ icp project bundle --output bundle.tar.gz
exit=0
$ tar tzvf bundle.tar.gz
-rw-r--r-- 0/0             973 1970-01-01 00:00 icp.yaml
-rw-r--r-- 0/0             410 1970-01-01 00:00 vendor/lib/icp.yaml
-rw-r--r-- 0/0          205794 1970-01-01 00:00 canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
-rw-r--r-- 0/0          205801 1970-01-01 00:00 canisters/353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d.wasm
$ grep -B1 -A2 "pre-built" icp.yaml vendor/lib/icp.yaml
vendor/lib/icp.yaml:    - type: pre-built
vendor/lib/icp.yaml-      path: ../../canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
vendor/lib/icp.yaml-      sha256: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a
icp.yaml:    - type: pre-built
icp.yaml-      path: canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
icp.yaml-      sha256: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a
--
icp.yaml:    - type: pre-built
icp.yaml-      path: canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
icp.yaml-      sha256: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a
--
icp.yaml:    - type: pre-built
icp.yaml-      path: canisters/353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d.wasm
icp.yaml-      sha256: 353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d
$ icp network start demo-net --background
For background mode, network output will be redirected:
  stdout: /tmp/bundledemo.uCcj/proj/x/.icp/cache/networks/demo-net/network-launcher/stdout.log
  stderr: /tmp/bundledemo.uCcj/proj/x/.icp/cache/networks/demo-net/network-launcher/stderr.log
Network started on port 33745
Seeding ICP and cycles account balances
To stop the network, run `icp network stop`
exit=0
$ icp deploy -e demo-env
Building canisters:
Creating canisters:
Setting environment variables:
Applying canister settings:
Checking compatibility:
Installing canisters:
No canisters have sync steps configured
Created canister one with ID 4qggx-l3777-77775-aaaca-cai
Created canister two with ID 4caro-hl777-77775-aaaba-cai
Created canister three with ID 4xhad-gd777-77775-aaacq-cai
Created canister vendor/lib:lib with ID 4fbx2-kt777-77775-aaabq-cai
Deployed canisters:

Backends (Candid UI):
  one: http://4mc4g-43777-77775-aaaaa-cai.localhost:33745/?id=4qggx-l3777-77775-aaaca-cai
  two: http://4mc4g-43777-77775-aaaaa-cai.localhost:33745/?id=4caro-hl777-77775-aaaba-cai
  three: http://4mc4g-43777-77775-aaaaa-cai.localhost:33745/?id=4xhad-gd777-77775-aaacq-cai
  vendor/lib:lib: http://4mc4g-43777-77775-aaaaa-cai.localhost:33745/?id=4fbx2-kt777-77775-aaabq-cai
exit=0
$ icp canister call -e demo-env one greet '("bundle")'
("Hello, bundle!")
exit=0
$ icp canister call -e demo-env two greet '("bundle")'
("Hello, bundle!")
exit=0
$ icp canister call -e demo-env three greet '("bundle")'
("Hello, bundle!")
exit=0
$ icp canister call -e demo-env vendor/lib:lib greet '("bundle")'
("Hello, bundle!")
exit=0
Evidence: Adversarial: tampered store entry and standalone dependency
# tampered: overwrote canisters/353ef...wasm with other bytes
$ icp build -e demo-env three
Building canisters:
ERR ----- Failed to build canister 'three' -----
ERR 'checksum mismatch, expected: 353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d, actual: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a'
ERR [three] Build output:
ERR [three] Building: step 1 of 1 (pre-built):
ERR [three] path: canisters/353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d.wasm, sha: 353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d:
ERR [three] > Reading wasm: canisters/353ef0568ee7a6702d635425affb0f792f48702bf334f8878fd742b59531877d.wasm
ERR [three] > Verifying checksum
Error: Canister(s) ["three"] failed to build.
exit=1

# dependency copied out of the bundle, used on its own
$ (cd solo && icp build -e demo-env lib)
Building canisters:
ERR ----- Failed to build canister 'lib' -----
ERR 'failed to read wasm file at '../../canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm''
ERR [lib] Build output:
ERR [lib] Building: step 1 of 1 (pre-built):
ERR [lib] path: ../../canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm, sha: 17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a:
ERR [lib] > Reading wasm: ../../canisters/17a05e36278cd04c7ae6d3d3226c136267b9df7525a0657521405e22ec96be7a.wasm
Error: Canister(s) ["lib"] failed to build.
exit=1
Evidence: Targeted bundle integration tests
   Compiling tokio v1.54.1
   Compiling nix v0.31.3
   Compiling predicates-core v1.0.10
   Compiling cryptoki-sys v0.5.0
   Compiling float-cmp v0.10.0
   Compiling libloading v0.8.9
   Compiling normalize-line-endings v0.3.0
   Compiling assert_cmd v2.2.2
   Compiling termtree v0.5.1
   Compiling difflib v0.4.0
   Compiling secrecy v0.10.3
   Compiling serial_test_derive v3.5.0
   Compiling fslock v0.2.1
   Compiling wait-timeout v0.2.1
   Compiling test-tag v0.1.6
   Compiling send_ctrlc v0.6.0
   Compiling predicates-tree v1.0.13
   Compiling predicates v3.1.4
   Compiling serial_test v3.5.0
   Compiling cryptoki v0.12.1
   Compiling ctrlc v3.5.2
   Compiling candid_parser v0.3.2
   Compiling tokio-util v0.7.19
   Compiling tower v0.5.3
   Compiling tokio-rustls v0.26.5
   Compiling icp-events v0.1.0 ({run-worktree}/crates/icp-events)
   Compiling wasmtime-wasi v49.0.5
   Compiling async-scoped v0.9.0
   Compiling backoff v0.4.0
   Compiling async-dropper-simple v0.2.6
   Compiling async-dropper v0.3.1
   Compiling tower-http v0.6.11
   Compiling h2 v0.4.19
   Compiling icp-project v0.1.0 ({run-worktree}/crates/icp-project)
   Compiling hyper v1.11.1
   Compiling hyper-util v0.1.20
   Compiling hyper-rustls v0.27.10
   Compiling hyperlocal v0.9.1
   Compiling httptest v0.16.4
   Compiling axum v0.8.9
   Compiling bollard v0.20.2
   Compiling reqwest v0.13.5
   Compiling ic-agent v0.49.2
   Compiling axoasset v2.0.1
   Compiling axoupdater v0.10.2
   Compiling ic-utils v0.49.2
   Compiling ic-identity-hsm v0.49.2
   Compiling icp-extension v1.8.0 ({run-worktree}/crates/icp-extension)
   Compiling icp-sync-plugin v0.1.0 ({run-worktree}/crates/icp-sync-plugin)
   Compiling icp-app v1.8.0 ({run-worktree}/crates/icp-app)
   Compiling icp-cli v1.8.0 ({run-worktree}/crates/icp-cli)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 49.72s
     Running tests/bundle_tests.rs (target/debug/deps/bundle_tests-9802c356b97d3c91)

running 9 tests
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
test bundle_stores_each_canister_wasm_once ... ok
test bundle_sanitizes_canister_name_for_paths ... ok
test bundle_packages_plugin_sync_steps ... ok
test bundle_archives_aliased_plugin_paths_once ... ok
test bundle_stores_each_plugin_wasm_once ... ok
test bundle_rewrites_dependency_paths_to_archive_locations ... ok
test bundle_preserves_dependency_structure ... ok
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on build directory
   Compiling proc-macro2 v1.0.106
   Compiling unicode-ident v1.0.24
   Compiling quote v1.0.45
   Compiling version_check v0.9.5
   Compiling serde_core v1.0.228
   Compiling autocfg v1.5.0
   Compiling serde v1.0.228
   Compiling rustversion v1.0.22
   Compiling thiserror v1.0.69
   Compiling object v0.37.3
   Compiling find-msvc-tools v0.1.9
   Compiling shlex v1.3.0
   Compiling crc32fast v1.5.0
   Compiling memchr v2.8.0
   Compiling typenum v1.20.0
   Compiling syn v1.0.109
   Compiling anyhow v1.0.102
   Compiling either v1.15.0
   Compiling cfg-if v1.0.4
   Compiling paste v1.0.15
   Compiling lazy_static v1.5.0
   Compiling libc v0.2.185
   Compiling strsim v0.11.1
   Compiling cpufeatures v0.2.17
   Compiling ident_case v1.0.1
   Compiling typed-arena v2.0.2
   Compiling unicode-width v0.2.2
   Compiling heck v0.5.0
   Compiling data-encoding v2.10.0
   Compiling arrayvec v0.5.2
   Compiling byteorder v1.5.0
   Compiling thiserror v2.0.18
   Compiling leb128 v0.2.6
   Compiling hex v0.4.3
   Compiling ic-cdk-executor v2.0.0
   Compiling smallvec v1.15.1
   Compiling ic0 v1.0.1
   Compiling pin-project-lite v0.2.17
   Compiling cc v1.2.60
   Compiling generic-array v0.14.7
   Compiling slotmap v1.1.1
   Compiling num-traits v0.2.19
   Compiling pretty v0.12.5
   Compiling syn v2.0.117
   Compiling stacker v0.1.23
   Compiling num-integer v0.1.46
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.7
   Compiling digest v0.10.7
   Compiling sha2 v0.10.9
   Compiling serde_bytes v0.11.19
   Compiling ar_archive_writer v0.5.1
   Compiling psm v0.1.30
   Compiling binread_derive v2.1.0
   Compiling darling_core v0.23.0
   Compiling serde_derive v1.0.228
   Compiling thiserror-impl v1.0.69
   Compiling candid_derive v0.10.27
   Compiling strum_macros v0.26.4
   Compiling thiserror-impl v2.0.18
   Compiling binread v2.2.0
   Compiling strum v0.26.3
   Compiling darling_macro v0.23.0
   Compiling darling v0.23.0
   Compiling num-bigint v0.4.6
   Compiling ic_principal v0.1.2
   Compiling ic-error-types v0.2.0
   Compiling candid v0.10.27
   Compiling ic-cdk-macros v0.20.0
   Compiling ic-cdk v0.20.0
   Compiling canister v0.1.0 ({run-worktree}/examples/icp-sync-plugin/canister)
    Finished `release` profile [optimized] target(s) in 13.97s
    Finished `release` profile [optimized] target(s) in 13.99s
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on build directory
   Compiling serde_core v1.0.228
   Compiling hashbrown v0.17.0
   Compiling equivalent v1.0.2
   Compiling foldhash v0.2.0
   Compiling zmij v1.0.21
   Compiling semver v1.0.28
   Compiling bitflags v2.11.1
   Compiling serde v1.0.228
   Compiling serde_json v1.0.149
   Compiling prettyplease v0.2.37
   Compiling typenum v1.20.0
   Compiling itoa v1.0.18
   Compiling unicode-xid v0.2.6
   Compiling leb128fmt v0.1.0
   Compiling log v0.4.29
   Compiling id-arena v2.3.0
   Compiling cfg-if v1.0.4
   Compiling wit-bindgen-rust v0.56.0
   Compiling wit-bindgen-rust-macro v0.56.0
   Compiling anyhow v1.0.102
   Compiling typed-arena v2.0.2
   Compiling lazy_static v1.5.0
   Compiling data-encoding v2.10.0
   Compiling arrayvec v0.5.2
   Compiling unicode-width v0.2.2
   Compiling wit-bindgen v0.56.0
   Compiling plugin v0.1.0 ({run-worktree}/examples/icp-sync-plugin/plugin)
   Compiling hex v0.4.3
   Compiling leb128 v0.2.6
   Compiling byteorder v1.5.0
   Compiling syn v2.0.117
   Compiling generic-array v0.14.7
   Compiling num-traits v0.2.19
   Compiling crc32fast v1.5.0
   Compiling thiserror v1.0.69
   Compiling hashbrown v0.16.1
   Compiling binread v2.2.0
   Compiling pretty v0.12.5
   Compiling num-integer v0.1.46
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.7
   Compiling digest v0.10.7
   Compiling sha2 v0.10.9
   Compiling indexmap v2.14.0
   Compiling serde_bytes v0.11.19
   Compiling macro-string v0.2.0
   Compiling serde_derive v1.0.228
   Compiling thiserror-impl v1.0.69
   Compiling candid_derive v0.10.27
   Compiling wasmparser v0.246.2
   Compiling num-bigint v0.4.6
   Compiling ic_principal v0.1.2
   Compiling candid v0.10.27
   Compiling wit-parser v0.246.2
   Compiling wasm-encoder v0.246.2
   Compiling wasm-metadata v0.246.2
   Compiling wit-bindgen-core v0.56.0
   Compiling wit-component v0.246.2
    Finished `release` profile [optimized] target(s) in 13.29s
    Finished `release` profile [optimized] target(s) in 13.31s
icp collects anonymous usage data to improve the tool.
Run `icp settings telemetry false` or set DO_NOT_TRACK=1 to opt out.
Learn more: https://github.com/dfinity/icp-cli/blob/v1.8.0/docs/telemetry.md
icp collects anonymous usage data to improve the tool.
Run `icp settings telemetry false` or set DO_NOT_TRACK=1 to opt out.
Learn more: https://github.com/dfinity/icp-cli/blob/v1.8.0/docs/telemetry.md
pocket-ic instance running with gateway port 39399
Seeding ICP and cycles account balances
pocket-ic instance running with gateway port 44607
Seeding ICP and cycles account balances
Network ready. Press Ctrl-C to exit.
Network ready. Press Ctrl-C to exit.
Received Ctrl-C, shutting down PocketIC...
Received Ctrl-C, shutting down PocketIC...
test bundle_with_dependency_deploys ... ok
test bundle_and_deploy ... ok

test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 35 filtered out; finished in 46.95s

EXIT 0
Evidence: Untrusted integration tests
   Compiling icp-cli v1.8.0 ({run-worktree}/crates/icp-cli)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 2.00s
     Running tests/untrusted_tests.rs (target/debug/deps/untrusted_tests-3398ddcdf4a6d99c)

running 11 tests
test a_bundle_shaped_project_builds_as_untrusted ... ok
test a_path_that_steps_back_out_of_a_symlink_is_refused_as_untrusted ... ok
test a_module_outside_the_project_is_refused_as_untrusted ... ok
test a_script_step_is_refused_as_untrusted_but_not_otherwise ... ok
test a_recipe_is_refused_as_untrusted ... ok
test a_file_read_from_outside_the_project_is_refused_as_untrusted ... ok
test a_relative_project_root_is_taken_from_the_working_directory ... ok
test a_bundle_member_reaches_its_wasms_at_the_bundle_root ... ok
test the_root_is_where_the_command_stands_and_is_not_searched_for ... ok
test the_flag_is_on_build_deploy_and_sync_alone ... ok
test a_symlink_is_refused_as_untrusted_wherever_it_leads ... ok

test result: ok. 11 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.65s

EXIT 0

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ℹ️ Cargo.lock:7065 - The libz-sys bump commit says no other dependencies changed, but tempfile's getrandom entry also moved from 0.4.3 to 0.3.4. tempfile 3.27 accepts either version and both are still in the lockfile, so nothing breaks. Only the commit message is inaccurate.
✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 6 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Bundling a project whose canisters (root and dependency) share a wasm stores one canister wasm per sha256 at canisters/<sha256>.wasm ✅ pass live manual_bundle_transcript.txt: tar listing shows only canisters/17a05e...wasm and canisters/353ef0...wasm for 4 canisters; also bundle_stores_each_canister_wasm_once
Root manifest names the store directly and the dependency manifest reaches it via ../.., each with a matching sha256 ✅ pass live manual_bundle_transcript.txt grep of pre-built steps; bundle_rewrites_dependency_paths_to_archive_locations
Extracted bundle deploys on a local network and every canister (including the dependency's) answers greet ✅ pass live manual_deploy_transcript.txt: deploy exit=0, all 4 greet calls return ("Hello, bundle!"); bundle_and_deploy and bundle_with_dependency_deploys passed
Identical plugin wasms used by several sync steps or instances are stored once at plugins/<sha256>.wasm, and the dependency's plugin runs on deploy ✅ pass live bundle_tests.log: bundle_stores_each_plugin_wasm_once, bundle_archives_aliased_plugin_paths_once, bundle_packages_plugin_sync_steps, bundle_with_dependency_deploys (dependency plugin seeds data during…
Adversarial: a tampered store entry is rejected by the sha256 check at build ✅ pass live manual_adversarial_transcript.txt: 'checksum mismatch, expected: 353ef..., actual: 17a05...' exit=1
Accepted tradeoff: a dependency copied out of the bundle can no longer build on its own; inside the bundle it builds as untrusted ✅ pass live manual_adversarial_transcript.txt: failed to read ../../canisters/<sha>.wasm, exit=1; untrusted_tests.log: a_bundle_member_reaches_its_wasms_at_the_bundle_root ok
  • cargo build --bin icp
  • cargo test -p icp-cli --test bundle_tests -- bundle_and_deploy bundle_with_dependency_deploys bundle_stores_each bundle_packages_plugin_sync_steps bundle_rewrites_dependency_paths bundle_preserves_dependency_structure bundle_archives_aliased bundle_sanitizes (9 passed)
  • cargo test -p icp-cli --test untrusted_tests (11 passed, incl. a_bundle_member_reaches_its_wasms_at_the_bundle_root and a_bundle_shaped_project_builds_as_untrusted)
  • Manual: icp project bundle on a disposable project with canisters one and two built from the same wasm, three from a distinct wasm, and a vendor/lib dependency also using the shared wasm; then tar tzvf and a check of the rewritten manifests
  • Manual: extracted the bundle, then icp network start --background, icp deploy -e demo-env, and icp canister call &lt;c&gt; greet for all 4 canisters against a local PocketIC launcher, using an isolated HOME and ICP_HOME
  • Manual adversarial: overwrote a store entry and ran icp build three, which failed with a checksum mismatch
  • Manual adversarial: copied the dependency out of the bundle and ran icp build lib, which failed because the wasm lives at the bundle root
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

…6>.wasm

A bundle wrote one copy of a plugin per (instance, canister, step), at
<instance>/plugins/<canister>/<idx>.wasm, so a workspace of several
canisters using the same plugin carried it many times over. Each distinct
plugin is now stored once at the archive root, named by the sha256 of the
bytes the bundler read; every step keeps its path: and sha256:, and a
dependency instance reaches the store with a path rising to the archive
root. Readers follow path: and verify sha256:, so none changes.

A dependency's directory in an extracted bundle therefore no longer works
on its own: the bundle is used from its root.
@raymondk
raymondk requested a review from a team as a code owner November 18, 2026 23:04
Copilot AI balanced review requested due to automatic review settings November 18, 2026 23:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the plugin deduplication change.

No open issues 🎉

What changed in this PR

This PR reduces project bundle size by storing each distinct plugin WebAssembly module once across all canisters and dependency instances, while preserving existing path: and sha256: fields.

Changes:

  • Stores plugins at plugins/<sha256>.wasm using a deduplicated, deterministically ordered map.
  • Rewrites dependency plugin references to the shared root store and documents that bundled dependencies are no longer standalone.
  • Adds deduplication and untrusted-loading coverage, and updates layout and deployment tests.
File Description
docs/​concepts/​project-dependencies.md Documents shared plugin storage and bundle-root usage requirements.
crates/​icp-project/​src/​operations/​bundle.rs Deduplicates plugins by content hash and generates relative references.
crates/​icp-cli/​tests/​untrusted_tests.rs Tests root loading and rejection of standalone dependency loading.
crates/​icp-cli/​tests/​bundle_tests.rs Verifies deduplication, updated archive paths, and manifest references.
CHANGELOG.md Records reduced duplication and the standalone-dependency limitation.

Review effort: Balanced

…ndencies:required` check. The `test:required` check needs no code change. - **dependencies:required:** cargo-deny failed on RUSTSEC-2026-0544, a newly published zlib advisory against `libz-sys` 1.1.31. That crate comes in through git2 and cargo-generate, not through this PR. The invariant: no dependency in the lockfile may match an unignored security advisory. The only place that holds it is `Cargo.lock`, so I ran `cargo update -p libz-sys`, which moved it from 1.1.31 to 1.1.33, the version the advisory recommends. No other dependencies changed. `cargo build --bin icp --locked` succeeds. cargo-deny is not installed here, so I could not re-run the advisory check locally; the next CI run will confirm it. The "advisory was not encountered" lines for the three zstd entries in `deny.toml` are warnings and don't fail the check, so I left them alone. - **test:required:** this is an aggregate job that runs `exit 1` when any test job didn't succeed. The only test job that didn't succeed was `canister_snapshot_tests on windows-2025`, and the provider cancelled it rather than it failing. That is infrastructure, not this PR's code. It needs a rerun, which is still awaiting a decision under ci-3
Copilot AI balanced review requested due to automatic review settings November 18, 2026 23:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The review identifies no active issues with the shared, content-addressed plugin storage change.

No open issues 🎉

Review effort: Balanced

…ha256>.wasm

Extend the plugin store to canister wasms. A bundle wrote one wasm per
canister at <instance>/canisters/<canister>.wasm, so canisters built
from the same module each carried a copy. Each distinct module is now
stored once at the archive root, named by the sha256 of the artifact
bytes; every pre-built step keeps its path: and sha256:, a dependency's
rising to the archive root. Readers follow path: and verify sha256:, so
none changes.

Same tradeoff as the plugin store: a member of an extracted bundle
resolves its wasm from the bundle root. An archive listing no longer
shows which canister a wasm belongs to; the manifests still do.
Copilot AI balanced review requested due to automatic review settings November 19, 2026 00:52
@raymondk raymondk changed the title feat(bundle): store each distinct plugin wasm once per bundle feat(bundle): store each distinct canister and plugin wasm once per bundle Nov 19, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the shared Wasm deduplication changes.

No open issues 🎉

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants