Repository navigation
Open
1 / 11 of 1 issue completedDescription
This is an umbrella bug for tracking all the work to setup infrastructure for pre-building native binaries and using them.
- s390x builds
- Integrate prebuilt binaries into release process
- upload binaries to releases
- update bzl code to refer to uploaded urls
- upload prebuilt artifact manifest telling their toolchain setup information (e.g. exec compatibility)
- include tool manifest in the release archive
- upload checksums of prebuilt artifacts
- Exec platforms built for, roughly
{linux,windows,mac} x {x86,arm} x {64-bit}-
aarch64-apple-darwin -
aarch64-pc-windows-msvc -
aarch64-unknown-linux-musl -
armv7-unknown-linux-musleabihf -
i686-pc-windows-msvc -
i686-unknown-linux-musl -
powerpc64le-unknown-linux-musl -
riscv64gc-unknown-linux-musl -
x86_64-apple-darwin -
x86_64-pc-windows-msvc -
x86_64-unknown-linux-musl -
s390x
-
- Target platforms built for, roughly exec platforms + 32bit
- ...
- verify/test at presubmit time
- test
bzlmod_lockfileintegration test with prebuilt toolchains (TOOL_DEPS_DEV_ONLY = True, ensuring it does not use toolchains from source) in addition to from-source mode
- test
- Tools to prebuild and bundle as a monolithic binary
- exe zip maker (
//tools/zipapp:exe_zip_maker///crates/exe_zip_maker) - zipapp packager (
//tools/zipapp:zipper) - zipapp
__main__.pymaker (//tools/zipapp:zip_main_maker) - console script binary maker (
//python/private:py_console_script_gen_py) - pyc precompiler (
//tools/precompiler:precompiler) -
py_testmain validator (//python/private:py_test_main_validator) - wheel maker (
//tools:wheelmaker) - wheel dist directory copier (
//python/private:py_wheel_dist) - wheel filegroup extractor (
//python/private/whl_filegroup:extract_wheel_files) - venv shebang rewriter (
//python/private/pypi:venv_shebang_rewriter) - wheel
RECORDrewriter (//python/private/pypi:wheel_record_rewriter) - build data writer (
//python/private:build_data_writer)
- exe zip maker (
- Binaries that are standalone
- precompilers: one per
{python_version, exec platform} - native bootstrap: one per target platform
- native zip exe bootstrap: one per target platform
- precompilers: one per
- Research/evaluate: bundling of tools (one monolithic tool or many small tools). Many have very small core logic, and their size is likely dominated by the rust language runtime than the core logic.
- Decision: monolithic tool (rationale)
- Figure out answer to how to handle non-prebuilt platforms
- Research/evaluation: Add a
--build_tools_from=<source|prebuilt>flag (name TBD) to control if the tool comes from source or from a prebuilt URL
Metadata
Metadata
Assignees
Labels
No labels
Activity
build(release): upload prebuilt tools and manifest to GitHub release
build(release): upload release artifacts and checksums to GitHub rele…
refactor(toolchains): configure prebuilt and from-source tool toolchains
refactor(toolchains): configure prebuilt and from-source tool toolcha…
tests: add integration test for prebuilt exe_zip_maker toolchain (#4309)
build(release): upload artifacts_manifest.json to GitHub releases (#4390
rickeylev commentedon Nov 1, 2026
I think a monolithic binary is the best option.
The runner up option is domain-specific binaries (e.g. one for core rules, one for pypi stuff, one for zipapp stuff, etc), but I don't think the benefits outweigh the additional configuration/design.
We have about 12 different tools and build for about 12 platforms. This would translate to 144 binaries for each release. Each one is about 500kb, which is mostly just the language runtime; the core logic of them is quite small. Separate binaries would also mean a separate bazel repo (and thus downloads) for each (we could bundle multiple into a single repo, but why?). It also means separate toolchains for each binary. I don't think any of that is a particularly large technical problem, but it seems like unnecessary overhead and unnecessary complication for the design.
In comparison, a single tool is about 1MB in size. It'd be a single repo, a single download, and a single toolchain. The downsize is any change to a build tool means all build tools are changed, so effectively all downstream targets are affected, so there's a big loss to incremental build performance. However, this would only affect environments that are building everything from source and regularly updating rules_python (or its dependencies), which I think is fairly rare, so I don't think its worth trying to optimize this case unless requested. (it wouldn't be too hard; we just have to have a way for rules to refer to different targets for the tools instead of all of them using the same target; i.e. separate toolchain types or a provide with separate attributes for the different tools. Under the hood, each tool is a separate crate that can be built into its own binary).
refactor(toolchains): generalize exe_zip_maker into rupyru-box toolchain
build(rust): add zip, flate2, and regex crates and upgrade sha2
refactor(zipapp): convert zipper build tool to Rust
refactor(zipapp): convert zip_main_maker build tool to Rust
refactor(entry_points): convert py_console_script_gen build tool to Rust
refactor: convert py_test_main_validator build tool to Rust
refactor(wheel): convert wheelmaker build tool to Rust
45 remaining items
refactor(whl_filegroup): convert extract_wheel_files build tool to Rust
refactor(pypi): convert venv_shebang_rewriter build tool to Rust
refactor(pypi): convert wheel_record_rewriter build tool to Rust
refactor: convert build_data_writer build tool to Rust
tests: add end-to-end test suite for Rust exec tools
[-]Setup prebuilt binaries infrastructure[/-][+]Setup prebuilt binaries infrastructure (get shrubbery)[/+]tests: add end-to-end harness for Rust exec tools
refactor(wheel): convert py_wheel_dist build tool to Rust
refactor(entry_points): convert py_console_script_gen build tool to Rust
refactor(whl_filegroup): convert extract_wheel_files build tool to Rust
refactor(pypi): convert venv_shebang_rewriter build tool to Rust
refactor(pypi): convert wheel_record_rewriter build tool to Rust
refactor: convert build_data_writer build tool to Rust
refactor: convert py_test_main_validator build tool to Rust
refactor(wheel): convert py_wheel_dist build tool to Rust (bazel-cont…