Repository navigation
Conversation
…6>.wasm A bundle wrote one copy of a plugin per (instance, canister, step), at <instance>/plugins/<canister>/<idx>.wasm, so a workspace of several canisters using the same plugin carried it many times over. Each distinct plugin is now stored once at the archive root, named by the sha256 of the bytes the bundler read; every step keeps its path: and sha256:, and a dependency instance reaches the store with a path rising to the archive root. Readers follow path: and verify sha256:, so none changes. A dependency's directory in an extracted bundle therefore no longer works on its own: the bundle is used from its root.
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the plugin deduplication change.
No open issues 🎉
What changed in this PR
This PR reduces project bundle size by storing each distinct plugin WebAssembly module once across all canisters and dependency instances, while preserving existing path: and sha256: fields.
Changes:
- Stores plugins at
plugins/<sha256>.wasmusing a deduplicated, deterministically ordered map. - Rewrites dependency plugin references to the shared root store and documents that bundled dependencies are no longer standalone.
- Adds deduplication and untrusted-loading coverage, and updates layout and deployment tests.
| File | Description |
|---|---|
docs/concepts/project-dependencies.md |
Documents shared plugin storage and bundle-root usage requirements. |
crates/icp-project/src/operations/bundle.rs |
Deduplicates plugins by content hash and generates relative references. |
crates/icp-cli/tests/untrusted_tests.rs |
Tests root loading and rejection of standalone dependency loading. |
crates/icp-cli/tests/bundle_tests.rs |
Verifies deduplication, updated archive paths, and manifest references. |
CHANGELOG.md |
Records reduced duplication and the standalone-dependency limitation. |
Review effort: Balanced
…ndencies:required` check. The `test:required` check needs no code change. - **dependencies:required:** cargo-deny failed on RUSTSEC-2026-0544, a newly published zlib advisory against `libz-sys` 1.1.31. That crate comes in through git2 and cargo-generate, not through this PR. The invariant: no dependency in the lockfile may match an unignored security advisory. The only place that holds it is `Cargo.lock`, so I ran `cargo update -p libz-sys`, which moved it from 1.1.31 to 1.1.33, the version the advisory recommends. No other dependencies changed. `cargo build --bin icp --locked` succeeds. cargo-deny is not installed here, so I could not re-run the advisory check locally; the next CI run will confirm it. The "advisory was not encountered" lines for the three zstd entries in `deny.toml` are warnings and don't fail the check, so I left them alone. - **test:required:** this is an aggregate job that runs `exit 1` when any test job didn't succeed. The only test job that didn't succeed was `canister_snapshot_tests on windows-2025`, and the provider cancelled it rather than it failing. That is infrastructure, not this PR's code. It needs a rerun, which is still awaiting a decision under ci-3
…ha256>.wasm Extend the plugin store to canister wasms. A bundle wrote one wasm per canister at <instance>/canisters/<canister>.wasm, so canisters built from the same module each carried a copy. Each distinct module is now stored once at the archive root, named by the sha256 of the artifact bytes; every pre-built step keeps its path: and sha256:, a dependency's rising to the archive root. Readers follow path: and verify sha256:, so none changes. Same tradeoff as the plugin store: a member of an extracted bundle resolves its wasm from the bundle root. An archive listing no longer shows which canister a wasm belongs to; the manifests still do.
lwshang
approved these changes
Nov 19, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Improve bundling in dfinity/icp-cli: today, when a project is bundled, the plugin for a wasm is potentially duplicated multiple times. Instead, the bundle should have a single copy per wasm hash in the entire bundle.
A scout explored this and proposed a plan (full report: /hoststorage/dev/firstmate/data/icp-cli-bundle-dedup-plan-f2/report.md). Substance: today
prepare_plugin_stepin crates/icp-project/src/operations/bundle.rs writes one copy per (instance, canister, step) at<instance>/plugins/<canister>/<idx>.wasm, so identical plugins are stored repeatedly (a demo with four asset canisters held five identical 3.6 MB copies). The plan stores each distinct plugin once at the archive root asplugins/<sha256>.wasm; each step keeps itspath:andsha256:fields, with a dependency instance reaching the store via a relative../..path. Readers (icp CLI trusted and --untrusted, imcp2-deploy, open-cloud deployer) followpath:and verifysha256:, so no reader changes. The one behaviour change: a dependency inside an extracted bundle no longer works on its own (as an untrusted root or copied out), because its plugin lives at the bundle root.The tradeoff for the subprojects not working outside of the root is acceptable for bundles - open a new PR against icp cli with this change.
We can actually expand the plugin dedup to also dedup canister wasms. This is more efficient if canisters in the same project are using the same wasm. Look into the feasibility and if it's doable amend the PR to include that as well.
What Changed
icp project bundlenow writes each distinct wasm only once for the whole archive. It is stored at the archive root and named by the SHA-256 of its bytes: canister wasms go tocanisters/<sha256>.wasm, which replaces the per-canister<instance>/canisters/<canister>.wasm, and plugin wasms go toplugins/<sha256>.wasm, which replaces the per-step<instance>/plugins/<canister>/<idx>.wasm. Incrates/icp-project/src/operations/bundle.rs, the artifact lists are nowBTreeMaps keyed by store path. This removes duplicate copies and keeps the entry order deterministic.path:andsha256:, so tools that read bundles need no change. A dependency instance reaches the shared store through a relative path that rises to the archive root, for example../../canisters/<sha256>.wasm. One behaviour changes as a result: a dependency's directory in an extracted bundle no longer works on its own, whether it is copied out or used as the root of--untrusted. The bundle has to be used from its root. The CHANGELOG anddocs/concepts/project-dependencies.mddescribe the new layout and this limitation.bundle_tests.rsanduntrusted_tests.rsfor the deduplicated layout and the shared-store references. Also bumpslibz-sys1.1.31 → 1.1.33 inCargo.lockto clear the RUSTSEC-2026-0544 advisory.🤖 Generated with Claude Code
Risk Assessment
✅ Low: This is a small, well-bounded layout change. Wasm modules are now stored under their own sha256 in BTreeMaps, so the dedup and the archive order are both deterministic. Every pre-built and plugin step still carries
path:andsha256:, a dependency reaches the root store through a correct relative../..path, the existing collision check still covers the shared store directories, and the intended tradeoff (a member no longer works outside the bundle root) is documented and tested.Testing
I built the realicpbinary and downloaded the pinned network launcher into the worktree. With them I bundled a disposable four-canister project that includes a dependency. The archive holds exactly two canister wasms, one per distinct hash. The root manifest names them ascanisters/<sha>.wasmand the dependency names its wasm as../../canisters/<sha>.wasm. The extracted bundle deployed on a local PocketIC network, and all four canisters answeredgreet. A tampered store file was rejected by the sha256 check. A dependency copied out of the bundle failed to build because its wasm lives at the bundle root, which is the tradeoff the user accepted. The targeted bundle and untrusted integration tests passed, including deploy-from-bundle with a plugin sync step in a dependency and plugin-wasm dedup. I removed the disposable project, the local network and the build outputs. Only the evidence transcripts remain.Evidence: Bundle listing and rewritten manifests (4 canisters, 2 distinct wasms → 2 store entries)
Evidence: Deploy from extracted bundle on local network + greet calls
Evidence: Adversarial: tampered store entry and standalone dependency
Evidence: Targeted bundle integration tests
Evidence: Untrusted integration tests
Pipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
Cargo.lock:7065- The libz-sys bump commit says no other dependencies changed, buttempfile'sgetrandomentry also moved from 0.4.3 to 0.3.4. tempfile 3.27 accepts either version and both are still in the lockfile, so nothing breaks. Only the commit message is inaccurate.✅ **Test** - passed
✅ No issues found.
cargo build --bin icpcargo test -p icp-cli --test bundle_tests -- bundle_and_deploy bundle_with_dependency_deploys bundle_stores_each bundle_packages_plugin_sync_steps bundle_rewrites_dependency_paths bundle_preserves_dependency_structure bundle_archives_aliased bundle_sanitizes(9 passed)cargo test -p icp-cli --test untrusted_tests(11 passed, incl. a_bundle_member_reaches_its_wasms_at_the_bundle_root and a_bundle_shaped_project_builds_as_untrusted)Manual:icp project bundleon a disposable project with canisters one and two built from the same wasm, three from a distinct wasm, and a vendor/lib dependency also using the shared wasm; thentar tzvfand a check of the rewritten manifestsManual: extracted the bundle, thenicp network start --background,icp deploy -e demo-env, andicp canister call <c> greetfor all 4 canisters against a local PocketIC launcher, using an isolated HOME and ICP_HOMEManual adversarial: overwrote a store entry and ranicp build three, which failed with a checksum mismatchManual adversarial: copied the dependency out of the bundle and ranicp build lib, which failed because the wasm lives at the bundle root✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.