Repository navigation
Open
Description
When I try to run bazel build @rules_python//python/config_settings:venvs_site_packages=yes //... I see the following output from one of my internal usages of py_wheel:
INFO: Invocation ID: 45e6114d-6e00-443c-95d4-ae1757315abd
ERROR: /Users/frank/workspace/yobi/data-alt-2/tools/runnable/examples/python/BUILD:10:17: Building wheel @@//tools/runnable/examples/python:example_app_deploy_wheel failed: (Exit 1): sandbox-exec failed: error executing PyWheel command
(cd /private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main && \
exec env - \
PATH=/bin:/usr/bin:/usr/local/bin \
TMPDIR=/var/folders/kb/_rxq15gs3gq21765gfcbbdjw0000gn/T/ \
/usr/bin/sandbox-exec -f /private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/sandbox.sb /var/tmp/_bazel_fp/install/f96a4af1851d4fb5dafa59d311169545/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/stats.out' bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker --name example_app --version 0 --python_tag py3 --abi none --platform any --out bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app-0-py3-none-any.whl --name_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel.name --input_file_list bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel_target_wrapped_inputs.txt --metadata_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel.metadata.txt --entry_points_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel_entry_points.txt)
Traceback (most recent call last):
File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 506, in <module>
main()
File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 500, in main
_run_py_path(main_filename, args=sys.argv[1:])
File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 287, in _run_py_path
runpy.run_path(main_filename, run_name="__main__")
File "<frozen runpy>", line 286, in run_path
File "<frozen runpy>", line 98, in _run_module_code
File "<frozen runpy>", line 88, in _run_code
File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 660, in <module>
main()
File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 560, in main
maker.add_file(package_filename, real_filename)
File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 323, in add_file
self._whlfile.add_file(package_filename, real_filename)
File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 173, in add_file
with open(real_filename, "rb") as fsrc:
^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/_example_app_bin.venv/lib/python3.12/site-packages/attr'
INFO: Elapsed time: 2.497s, Critical Path: 2.02s
INFO: 79 processes: 53 action cache hit, 79 internal.
ERROR: Build did NOT complete successfully
I am still investigating whether this is user error and whether I may have been doing something "unofficial" without declaring formal runfiles deps, but wanted to post this early since I know there is a lot of ongoing work for this feature.
Metadata
Metadata
Assignees
Labels
No labels
Activity
rickeylev commentedon Dec 19, 2025
Are you putting a py_binary in a py_wheel? It's dubious if that's supported or not.
This sounds similar to the rules_pkg issue where, because the venv uses relative symlinks, handling them needs to be done specially.
In this case, it looks like the wheel packager is trying to read a symlink to a directory as if it was a file. Zips don't officially support storing symlinks, but it's possible with some special extensions. At the least, it could dereference the symlinks at do a full copy.
FrankPortman commentedon Dec 19, 2025
Yes - for legacy reasons. And in practice I only care about the
PyInfo-esque information. I had to introduce a different heuristic to usebootstrap_impl=script(related to when I opened: #3324).I think if my rule is working then it shouldn't even try to copy external files though (it should separate out external packages into tuples of
(package, version). I will need to look into this more.shayanhoshyari commentedon Dec 22, 2025
@rickeylev I am facing a different instance of Missing files when running with
venvs_site_packages=yeswith1.7.0 + linux.TLDR is that, when I do
bazel-bin/app.runfiles/_main/_app.venv/lib/python3.13/site-packages/nvidia/nccl//lib/libnccl.so.2bazel-bin/app.runfiles/rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/nvidia/nccl/lib/libnccl.so.2is included..sofiles seem to be where they should be in th venv, e.g.bazel-bin/app.runfiles/_main/_app.venv/lib/python3.13/site-packages/nvidia/cufft/lib/libcufft.so.11If I do however
the
.soappears. If I do both torch and nccl_cu12, the .so goes missing again.Here is a full repro: https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/missing_so
What this causes is that if my docker container does not have
ncclthen import torch completely fails. If it does, but different version, importing torch gives things like this due to version mismatchSome debugging with this modified code
Debug prints: https://github.com/bazel-contrib/rules_python/compare/1.7.0...shayanhoshyari:rules_python:hoshyari/issue-3470?expand=1
printed output
group 1case. https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L117-L122VenvSymlinkEntryfor them. Atstartswithfailing here: https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L318-L330This is on 1.7.0. I see there are changes in these areas in main, so this analysis might be stale there. That being said, why in main I am losing all .so s, that is another question.
shayanhoshyari commentedon Dec 22, 2025
I tried the head, 9a20fc243a23399511023d78209590f776b36e38 of rules_python with same code, suddenly all .so symlinks went missing :|
rickeylev commentedon Dec 22, 2025
Great debugging, thank you! I'll have a look. I was manually testing torch + nvidia deps, so hrmph 😅 . I swear a added a unit test exactly for this combination of things >.<
link_to_path = [...]/site-packages/libnccl.so.2-- becausenvidia/nccl/libis missing in the middle, it probably means something is wrong in the get_venv_symlinks() function. Somehow around L342, wherelib*.so*files are handled. Its the only place where an entry like that gets created (one with link_to_file set)shayanhoshyari commentedon Dec 22, 2025
@rickeylev Thanks for quick reply. I edited my post #3470 (comment) with the prints I added, and some more info.
You mean https://github.com/bazel-contrib/rules_python/blob/main/tests/venv_site_packages_libs/shared_lib_loading_test.py right? Looking here a few questions:
1.7.0
1.7.0, is it intentional that you are makingadder.so(withoutlib) here andlibincrement.so? but the check here explicitly looks forlibprefix? I found this out when seeing this entry making does not apply to adder and instead it is saved by this one.1.7.0I can make the unit test break (emulating by what happens to nvidia/nccl) with this change: https://github.com/bazel-contrib/rules_python/compare/1.7.0...shayanhoshyari:rules_python:hoshyari/issue-3470?expand=1 (the changes in venv_runfiles.bzl is just prints, the failure changes are simply in the test BUILD).In general in
1.7.0the logic around adding entries multiple times for same.sofile, the second one coming from https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L318-L330 makes things hard to debug as somethings pass luckily due to second entry (but not always). E.g. the unit test.But in theory 41f91e9 (in main and not in
1.7.0should address the above. Now let me move tomainand see what is going on there. I do see missing stuff there too: https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/missing_so#mainmain
😂 ok this was an easy one. issue was here: https://github.com/bazel-contrib/rules_python/blob/main/python/private/venv_runfiles.bzl#L67. when
ctx.packageis "", i.e. root BUILD, you get an abs path, and messes things up.Change
to
fixes it.
Let me raise a PR
shayanhoshyari commentedon Dec 22, 2025
@rickeylev fix for my issue in main (i.e. #3470 (comment)): #3474
fix(venv): Fix all .so files missing when py_binary lives at //:BUILD (…
rickeylev commentedon Dec 22, 2025
Heh, yeah, the logic and code structure in 1.7 and earlier was convoluted. Nice to have that refactor validated 😄 .
Yes. It's verifying that
foo.so(python C modules, not regular C libs) don't get explicitly linked (and thus can be grouped into fewer link operations). From what I could tell, it was onlylib*.so*files being processed by the dynamic linker that would compute the wrong LD search path. Python C modules (loaded via python's import system) didn't seem to exhibit that same behavior. LMK if you see otherwise.FrankPortman commentedon Dec 22, 2025
@rickeylev re my question in the initial issue - is this impacted by the recent PR in the thread and/or is the missing file WAI?
rickeylev commentedon Dec 23, 2025
Oh whoops, no, the thing shayanhashyari fixed is something different. I'll reopen this.
kwlzn commentedon Feb 2, 2026
I think I'm seeing the same (or a similar issue), but:
common --@rules_python//python/config_settings:venvs_site_packages=yespy_binarytargets break with missing files (6,000 files in one targets case) in runfiles referencing files in venv site-packages paths like:I'll see if I can work up a minimal repro.
BryceBeagle commentedon Feb 20, 2026
We're also seeing some similar errors relating to missing files. But it doesn't seem deterministic (most builds succeed), and is only happening in our CI (haven't been able to reproduce locally and dive into the filesystem state yet):
This was occurring very seldomly on
1.7.0but is now occurring much more often on1.8.4This is happening only for targets that are using
(were selectively applying this too some
py_console_script_binarys as we work to enable it globally)BryceBeagle commentedon Feb 20, 2026
One interesting observation:
We use persistent runners to share a bazel cache between multiple CI pipelines. Since we've merged the
1.8.4upgrade this error is actually occurring when our CI runners run tests for branches not yet rebased past the upgrade. It's like the1.8.4upgrade is poisoning the build tree for other CI executions.