Skip to content

Missing files when running with venvs_site_packages=yes #3470

Open

Description

@FrankPortman

When I try to run bazel build @rules_python//python/config_settings:venvs_site_packages=yes //... I see the following output from one of my internal usages of py_wheel:

INFO: Invocation ID: 45e6114d-6e00-443c-95d4-ae1757315abd
ERROR: /Users/frank/workspace/yobi/data-alt-2/tools/runnable/examples/python/BUILD:10:17: Building wheel @@//tools/runnable/examples/python:example_app_deploy_wheel failed: (Exit 1): sandbox-exec failed: error executing PyWheel command
  (cd /private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main && \
  exec env - \
    PATH=/bin:/usr/bin:/usr/local/bin \
    TMPDIR=/var/folders/kb/_rxq15gs3gq21765gfcbbdjw0000gn/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/sandbox.sb /var/tmp/_bazel_fp/install/f96a4af1851d4fb5dafa59d311169545/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/stats.out' bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker --name example_app --version 0 --python_tag py3 --abi none --platform any --out bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app-0-py3-none-any.whl --name_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel.name --input_file_list bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel_target_wrapped_inputs.txt --metadata_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel.metadata.txt --entry_points_file bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/example_app_deploy_wheel_entry_points.txt)
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 506, in <module>
    main()
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 500, in main
    _run_py_path(main_filename, args=sys.argv[1:])
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/_main/../rules_python+/tools/_wheelmaker_stage2_bootstrap.py", line 287, in _run_py_path
    runpy.run_path(main_filename, run_name="__main__")
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 660, in <module>
    main()
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 560, in main
    maker.add_file(package_filename, real_filename)
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 323, in add_file
    self._whlfile.add_file(package_filename, real_filename)
  File "/private/var/tmp/_bazel_fp/01684213bb9c693047272ec8f0247481/sandbox/darwin-sandbox/5853/execroot/_main/bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/external/rules_python+/tools/wheelmaker.runfiles/rules_python+/tools/wheelmaker.py", line 173, in add_file
    with open(real_filename, "rb") as fsrc:
         ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'bazel-out/darwin_arm64-fastbuild/bin/tools/runnable/examples/python/_example_app_bin.venv/lib/python3.12/site-packages/attr'
INFO: Elapsed time: 2.497s, Critical Path: 2.02s
INFO: 79 processes: 53 action cache hit, 79 internal.
ERROR: Build did NOT complete successfully

I am still investigating whether this is user error and whether I may have been doing something "unofficial" without declaring formal runfiles deps, but wanted to post this early since I know there is a lot of ongoing work for this feature.

Activity

  1. rickeylev commented on Dec 19, 2025

    @rickeylev
    Collaborator

    Are you putting a py_binary in a py_wheel? It's dubious if that's supported or not.

    This sounds similar to the rules_pkg issue where, because the venv uses relative symlinks, handling them needs to be done specially.

    In this case, it looks like the wheel packager is trying to read a symlink to a directory as if it was a file. Zips don't officially support storing symlinks, but it's possible with some special extensions. At the least, it could dereference the symlinks at do a full copy.

  2. FrankPortman commented on Dec 19, 2025

    @FrankPortman
    ContributorAuthor

    Are you putting a py_binary in a py_wheel?

    Yes - for legacy reasons. And in practice I only care about the PyInfo-esque information. I had to introduce a different heuristic to use bootstrap_impl=script (related to when I opened: #3324).

    def _is_runtime_artifact(file):
        path = file.short_path
        basename = file.basename
    
        # Venv artifacts added by `rules_python` with bootstrap_impl=script
        if path.endswith(".venv/bin/python3"):
            return True
    
        # Bootstrap scripts added by `rules_python` with bootstrap_impl=script
        if basename.endswith("_stage2_bootstrap.py"):
            return True
    
        return False
    
        ...
    
        inputs = depset(
            transitive = [dep[DefaultInfo].data_runfiles.files for dep in ctx.attr.deps] +
                         [dep[DefaultInfo].default_runfiles.files for dep in ctx.attr.deps],
        )
    
        # Gather the pypi deps
        direct_pypi_deps = [dep for dep in ctx.attr.deps if PypiInfo in dep]
        indirect_deps = [dep[PypiInfo].pypi_deps for dep in direct_pypi_deps]
        all_deps = depset(direct = direct_pypi_deps, transitive = indirect_deps)
    
        infos = [
            (dep[PypiInfo].name, dep[PypiInfo].version)
            for dep in all_deps.to_list()
            if hasattr(dep[PypiInfo], "name")
        ]
    
        internal_deps = [
            dep
            for dep in inputs.to_list()
            if dep.owner.repo_name == _THIS_REPO and not _is_runtime_artifact(dep)
        ]
    
        ...

    I think if my rule is working then it shouldn't even try to copy external files though (it should separate out external packages into tuples of (package, version). I will need to look into this more.

  3. shayanhoshyari commented on Dec 22, 2025

    @shayanhoshyari
    Contributor

    @rickeylev I am facing a different instance of Missing files when running with venvs_site_packages=yes with 1.7.0 + linux.

    TLDR is that, when I do

    py_binary(
        name = "hello_world_app",
        srcs = [
            "app.py",
            "__main__.py"
        ],
        main = "__main__.py",
        deps = [
            "@pip//cowsay",
            "@pip//torch",
        ],
    )
    • At least one .so is missing bazel-bin/app.runfiles/_main/_app.venv/lib/python3.13/site-packages/nvidia/nccl//lib/libnccl.so.2
    • While the symlink is missing, the root runfile bazel-bin/app.runfiles/rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/nvidia/nccl/lib/libnccl.so.2 is included.
    • the other .py files from this library are where they should be).
    • Other .so files seem to be where they should be in th venv, e.g. bazel-bin/app.runfiles/_main/_app.venv/lib/python3.13/site-packages/nvidia/cufft/lib/libcufft.so.11

    If I do however

    py_binary(
        name = "hello_world_app",
        srcs = [
            "app.py",
            "__main__.py"
        ],
        main = "__main__.py",
        deps = [
            "@pip//cowsay",
            "@pip//nvidia_nccl_cu12",
        ],
    )

    the .so appears. If I do both torch and nccl_cu12, the .so goes missing again.

    Here is a full repro: https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/missing_so

    What this causes is that if my docker container does not have nccl then import torch completely fails. If it does, but different version, importing torch gives things like this due to version mismatch

        from torch._C import *  # noqa: F403
        ^^^^^^^^^^^^^^^^^^^^^^
    undefined symbol: ncclGroupSimulateEnd
    

    Some debugging with this modified code

    Debug prints: https://github.com/bazel-contrib/rules_python/compare/1.7.0...shayanhoshyari:rules_python:hoshyari/issue-3470?expand=1

    printed output
    Computing main repo mapping: 
    Loading: 
    Loading: 1 packages loaded
    Analyzing: target //:app (2 packages loaded, 0 targets configured)
    Analyzing: target //:app (2 packages loaded, 0 targets configured)
    
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:300:22: 
    	entry: struct(files = depset([<source file site-packages/nvidia/nccl/lib/libnccl.so.2>]), kind = "LIB", link_to_file = <source file site-packages/nvidia/nccl/lib/libnccl.so.2>, link_to_path = "rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/libnccl.so.2", package = "nvidia_nccl_cu12", venv_path = "nvidia/nccl/lib/libnccl.so.2", version = "2.27.5") 
    	runfiles_dir_name: rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1 
    	site_packages_root: site-packages/ 
    	filename: libnccl.so.2
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:300:22: 
    	entry: struct(files = depset([<source file site-packages/nvidia/cufft/lib/libcufft.so.11>]), kind = "LIB", link_to_file = <source file site-packages/nvidia/cufft/lib/libcufft.so.11>, link_to_path = "rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/libcufft.so.11", package = "nvidia_cufft_cu12", venv_path = "nvidia/cufft/lib/libcufft.so.11", version = "11.3.3.83") 
    	runfiles_dir_name: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e 
    	site_packages_root: site-packages/ 
    	filename: libcufft.so.11
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
    	group: cowsay 1
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
    	group: cowsay-6.1.dist-info 1
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
    	group: nvidia 7
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:221:22: 
    	 .so. WINS 
    	entry.files [<source file site-packages/nvidia/__init__.py>, <source file site-packages/nvidia/nvjitlink/__init__.py>, <source file site-packages/nvidia/nvjitlink/include/__init__.py>, <source file site-packages/nvidia/nvjitlink/include/nvJitLink.h>, <source file site-packages/nvidia/nvjitlink/lib/__init__.py>, <source file site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12>] 
    	entry.venv_path: nvidia 
    	rf_root_path: rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12 
    	entry.link_to_path: rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/nvidia
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:221:22: 
    	 .so. WINS 
    	entry.files [<source file site-packages/nvidia/__init__.py>, <source file site-packages/nvidia/cufft/__init__.py>, <source file site-packages/nvidia/cufft/include/__init__.py>, <source file site-packages/nvidia/cufft/include/cudalibxt.h>, <source file site-packages/nvidia/cufft/include/cufft.h>, <source file site-packages/nvidia/cufft/include/cufftXt.h>, <source file site-packages/nvidia/cufft/include/cufftw.h>, <source file site-packages/nvidia/cufft/lib/__init__.py>, <source file site-packages/nvidia/cufft/lib/libcufft.so.11>, <source file site-packages/nvidia/cufft/lib/libcufftw.so.11>] 
    	entry.venv_path: nvidia 
    	rf_root_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia/cufft/lib/libcufft.so.11 
    	entry.link_to_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:221:22: 
    	 .so. WINS 
    	entry.files [<source file site-packages/nvidia/__init__.py>, <source file site-packages/nvidia/cufft/__init__.py>, <source file site-packages/nvidia/cufft/include/__init__.py>, <source file site-packages/nvidia/cufft/include/cudalibxt.h>, <source file site-packages/nvidia/cufft/include/cufft.h>, <source file site-packages/nvidia/cufft/include/cufftXt.h>, <source file site-packages/nvidia/cufft/include/cufftw.h>, <source file site-packages/nvidia/cufft/lib/__init__.py>, <source file site-packages/nvidia/cufft/lib/libcufft.so.11>, <source file site-packages/nvidia/cufft/lib/libcufftw.so.11>] 
    	entry.venv_path: nvidia 
    	rf_root_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia/cufft/lib/libcufftw.so.11 
    	entry.link_to_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:203:26: 
    	 .so. loses
    	 entry.files [<source file site-packages/nvidia/cufft/lib/libcufft.so.11>] 
    	entry.venv_path: nvidia/cufft/lib/libcufft.so.11 
    	rf_root_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia/cufft/lib/libcufft.so.11 
    	entry.link_to_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/libcufft.so.11
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:203:26: 
    	 .so. loses
    	 entry.files [<source file site-packages/nvidia/cufft/lib/libcufftw.so.11>] 
    	entry.venv_path: nvidia/cufft/lib/libcufftw.so.11 
    	rf_root_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia/cufft/lib/libcufftw.so.11 
    	entry.link_to_path: rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/libcufftw.so.11
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:203:26: 
    	 .so. loses
    	 entry.files [<source file site-packages/nvidia/nccl/lib/libnccl.so.2>] 
    	entry.venv_path: nvidia/nccl/lib/libnccl.so.2 
    	rf_root_path: rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/nvidia/nccl/lib/libnccl.so.2 
    	entry.link_to_path: rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/libnccl.so.2
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:203:26: 
    	 .so. loses
    	 entry.files [<source file site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12>] 
    	entry.venv_path: nvidia/nvjitlink/lib/libnvJitLink.so.12 
    	rf_root_path: rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12 
    	entry.link_to_path: rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/libnvJitLink.so.12
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
    	group: nvidia_cufft_cu12-11.3.3.83.dist-info 1
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
    	group: nvidia_nccl_cu12-2.27.5.dist-info 1
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:120:18: 
    	group: nvidia_nvjitlink_cu12-12.8.93.dist-info 1
    DEBUG: /mnt/localssd/.cache/bazel/_bazel_colligo/afcdef9f6a839f43a157a9e747a64eed/external/rules_python+/python/private/venv_runfiles.bzl:132:14: 
    	keep_kind_link_map: {"cowsay": "rules_python++pip+pip_313_cowsay_py3_none_any_274b1e6f/site-packages/cowsay", "cowsay-6.1.dist-info": "rules_python++pip+pip_313_cowsay_py3_none_any_274b1e6f/site-packages/cowsay-6.1.dist-info", "nvidia/__init__.py": <source file site-packages/nvidia/__init__.py>, "nvidia/nvjitlink/__init__.py": <source file site-packages/nvidia/nvjitlink/__init__.py>, "nvidia/nvjitlink/include/__init__.py": <source file site-packages/nvidia/nvjitlink/include/__init__.py>, "nvidia/nvjitlink/include/nvJitLink.h": <source file site-packages/nvidia/nvjitlink/include/nvJitLink.h>, "nvidia/nvjitlink/lib/__init__.py": <source file site-packages/nvidia/nvjitlink/lib/__init__.py>, "nvidia/nvjitlink/lib/libnvJitLink.so.12": <source file site-packages/nvidia/nvjitlink/lib/libnvJitLink.so.12>, "nvidia/cufft/__init__.py": <source file site-packages/nvidia/cufft/__init__.py>, "nvidia/cufft/include/__init__.py": <source file site-packages/nvidia/cufft/include/__init__.py>, "nvidia/cufft/include/cudalibxt.h": <source file site-packages/nvidia/cufft/include/cudalibxt.h>, "nvidia/cufft/include/cufft.h": <source file site-packages/nvidia/cufft/include/cufft.h>, "nvidia/cufft/include/cufftXt.h": <source file site-packages/nvidia/cufft/include/cufftXt.h>, "nvidia/cufft/include/cufftw.h": <source file site-packages/nvidia/cufft/include/cufftw.h>, "nvidia/cufft/lib/__init__.py": <source file site-packages/nvidia/cufft/lib/__init__.py>, "nvidia/cufft/lib/libcufft.so.11": <source file site-packages/nvidia/cufft/lib/libcufft.so.11>, "nvidia/cufft/lib/libcufftw.so.11": <source file site-packages/nvidia/cufft/lib/libcufftw.so.11>, "nvidia/nccl/include/nccl.h": <source file site-packages/nvidia/nccl/include/nccl.h>, "nvidia_cufft_cu12-11.3.3.83.dist-info": "rules_python++pip+pip_313_nvidia_cufft_cu12_py3_none_manylinux_2_17_x86_64_4d2dd21e/site-packages/nvidia_cufft_cu12-11.3.3.83.dist-info", "nvidia_nccl_cu12-2.27.5.dist-info": "rules_python++pip+pip_313_nvidia_nccl_cu12_py3_none_manylinux_2_17_x86_64_ad730cf1/site-packages/nvidia_nccl_cu12-2.27.5.dist-info", "nvidia_nvjitlink_cu12-12.8.93.dist-info": "rules_python++pip+pip_313_nvidia_nvjitlink_cu12_py3_none_manylinux_2_12_x86_64_81ff6337/site-packages/nvidia_nvjitlink_cu12-12.8.93.dist-info"}
    INFO: Analyzed target //:app (90 packages loaded, 3391 targets configured).
    INFO: Found 1 target...
    Target //:app up-to-date:
      bazel-bin/app
    INFO: Elapsed time: 0.261s, Critical Path: 0.00s
    INFO: 1 process: 31 action cache hit, 1 internal.
    INFO: Build completed successfully, 1 total action
    
    
    1. only single nccl works because here we enter the group 1 case. https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L117-L122
    2. Other libs somehow work (e.g. CUFFT) because we add multiple VenvSymlinkEntry for them. At
    1. some libs like nccl only go to 2.1, and not 2.2 so their links never gets created.

    This is on 1.7.0. I see there are changes in these areas in main, so this analysis might be stale there. That being said, why in main I am losing all .so s, that is another question.

  4. shayanhoshyari commented on Dec 22, 2025

    @shayanhoshyari
    Contributor

    I tried the head, 9a20fc243a23399511023d78209590f776b36e38 of rules_python with same code, suddenly all .so symlinks went missing :|

  5. rickeylev commented on Dec 22, 2025

    @rickeylev
    Collaborator

    Great debugging, thank you! I'll have a look. I was manually testing torch + nvidia deps, so hrmph 😅 . I swear a added a unit test exactly for this combination of things >.<

    link_to_path = [...]/site-packages/libnccl.so.2 -- because nvidia/nccl/lib is missing in the middle, it probably means something is wrong in the get_venv_symlinks() function. Somehow around L342, where lib*.so* files are handled. Its the only place where an entry like that gets created (one with link_to_file set)

  6. shayanhoshyari commented on Dec 22, 2025

    @shayanhoshyari
    Contributor

    @rickeylev Thanks for quick reply. I edited my post #3470 (comment) with the prints I added, and some more info.

    I swear a added a unit test exactly for this combination of things

    You mean https://github.com/bazel-contrib/rules_python/blob/main/tests/venv_site_packages_libs/shared_lib_loading_test.py right? Looking here a few questions:

    1.7.0

    1. On 1.7.0, is it intentional that you are making adder.so (without lib) here and libincrement.so? but the check here explicitly looks for lib prefix? I found this out when seeing this entry making does not apply to adder and instead it is saved by this one.
    2. In 1.7.0 I can make the unit test break (emulating by what happens to nvidia/nccl) with this change: https://github.com/bazel-contrib/rules_python/compare/1.7.0...shayanhoshyari:rules_python:hoshyari/issue-3470?expand=1 (the changes in venv_runfiles.bzl is just prints, the failure changes are simply in the test BUILD).

    In general in 1.7.0 the logic around adding entries multiple times for same .so file, the second one coming from https://github.com/bazel-contrib/rules_python/blame/1.7.0/python/private/venv_runfiles.bzl#L318-L330 makes things hard to debug as somethings pass luckily due to second entry (but not always). E.g. the unit test.

    But in theory 41f91e9 (in main and not in 1.7.0 should address the above. Now let me move to main and see what is going on there. I do see missing stuff there too: https://github.com/shayanhoshyari/issue-reports/tree/main/rules_python/missing_so#main

    main

    😂 ok this was an easy one. issue was here: https://github.com/bazel-contrib/rules_python/blob/main/python/private/venv_runfiles.bzl#L67. when ctx.package is "", i.e. root BUILD, you get an abs path, and messes things up.

    Change

    symlink_from = "{}/{}".format(ctx.label.package, bin_venv_path)
    

    to

    paths.join(ctx.label.package, bin_venv_path) 
    

    fixes it.

    Let me raise a PR

  7. shayanhoshyari commented on Dec 22, 2025

    @shayanhoshyari
    Contributor

    @rickeylev fix for my issue in main (i.e. #3470 (comment)): #3474

  8. rickeylev commented on Dec 22, 2025

    @rickeylev
    Collaborator

    Heh, yeah, the logic and code structure in 1.7 and earlier was convoluted. Nice to have that refactor validated 😄 .

    is it intentional that you are making adder.so (without lib) here and libincrement.so?

    Yes. It's verifying that foo.so (python C modules, not regular C libs) don't get explicitly linked (and thus can be grouped into fewer link operations). From what I could tell, it was only lib*.so* files being processed by the dynamic linker that would compute the wrong LD search path. Python C modules (loaded via python's import system) didn't seem to exhibit that same behavior. LMK if you see otherwise.

  9. FrankPortman commented on Dec 22, 2025

    @FrankPortman
    ContributorAuthor

    @rickeylev re my question in the initial issue - is this impacted by the recent PR in the thread and/or is the missing file WAI?

  10. rickeylev commented on Dec 23, 2025

    @rickeylev
    Collaborator

    Oh whoops, no, the thing shayanhashyari fixed is something different. I'll reopen this.

  11. kwlzn commented on Feb 2, 2026

    @kwlzn

    I think I'm seeing the same (or a similar issue), but:

    1. after upgrading to rules_python 1.8.3 (or using latest main)
    2. with common --@rules_python//python/config_settings:venvs_site_packages=yes
    3. py_binary targets break with missing files (6,000 files in one targets case) in runfiles referencing files in venv site-packages paths like:
    bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/numpy/fft/helper.pyi: No such file or directory (os error 2)
    bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/numpy/fft/tests: No such file or directory (os error 2)
    bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/numpy/lib: No such file or directory (os error 2)
    bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/numpy/linalg/__init__.py: No such file or directory (os error 2)
    bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/numpy/linalg/__init__.pyi: No such file or directory (os error 2)
    bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/numpy/linalg/linalg.py: No such file or directory (os error 2)
    bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/numpy/linalg/linalg.pyi: No such file or directory (os error 2)
    
    1. the missing files are all dangling symlinks that look like e.g.:
    $ ls -al bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/PIL/CurImagePlugin.py
    lrwxrwxrwx 1 vscode vscode 96 Feb  2 01:45 bazel-out/k8-opt-release/bin/python_modules/m_app/data_types/_dataset_types.venv/lib/python3.10/site-packages/PIL/CurImagePlugin.py -> ../../../../../../../../../rules_python++pip+pypi_310_pillow/site-packages/PIL/CurImagePlugin.py
    

    I'll see if I can work up a minimal repro.

  12. BryceBeagle commented on Feb 20, 2026

    @BryceBeagle

    We're also seeing some similar errors relating to missing files. But it doesn't seem deterministic (most builds succeed), and is only happening in our CI (haven't been able to reproduce locally and dive into the filesystem state yet):

    ERROR: /<snip>/domains/rapid/smoke_tests/apps/apis/smoke_test_py_rayserve/BUILD.bazel:15:12:
    Creating symlink bazel-out/k8-opt-ST-c6a2b441b098/bin/domains/rapid/smoke_tests/apps/apis/smoke_test_py_rayserve/_ray_linux_x86_64.venv/lib/python3.12/site-packages/scripts/test_imports.py failed:
    failed to create symbolic link 'bazel-out/k8-opt-ST-c6a2b441b098/bin/domains/rapid/smoke_tests/apps/apis/smoke_test_py_rayserve/_ray_linux_x86_64.venv/lib/python3.12/site-packages/scripts/test_imports.py' with target '../../../../../../../../../../../../pypi_smoke_test_py_rayserve_linux_x86_64_pyarrow/site-packages/scripts/test_imports.py' due to I/O error:
    /data/output/execroot/dd_source/bazel-out/k8-opt-ST-c6a2b441b098/pypi_smoke_test_py_rayserve_linux_x86_64_pyarrow (No such file or directory)
    

    This was occurring very seldomly on 1.7.0 but is now occurring much more often on 1.8.4

    This is happening only for targets that are using

    config_settings = {"@rules_python//python/config_settings:venvs_site_packages" = "yes"}

    (were selectively applying this too some py_console_script_binarys as we work to enable it globally)

  13. BryceBeagle commented on Feb 20, 2026

    @BryceBeagle

    One interesting observation:

    We use persistent runners to share a bazel cache between multiple CI pipelines. Since we've merged the 1.8.4 upgrade this error is actually occurring when our CI runners run tests for branches not yet rebased past the upgrade. It's like the 1.8.4 upgrade is poisoning the build tree for other CI executions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions