Skip to content

Scope pip.override to single version of referenced wheel #2263

Closed

Description

@keith

In our project we pull in the same dependency through multiple pip.parse hubs, and they're used different places in our build graph. We want to apply a patch in 1 of those versions, but not the others. Based on the current API accepting a specific wheel name, I expected this to happen automatically, since at least in our case the wheel name includes the version etc, such that it wouldn't have any conflicts across the other downloads. I was surprised to see the other downloads start warning with RECORD mismatches because the patches are applying there too.

I propose that when using file with pip.override, only the specific matching filename is patched. And instead add a new attribute such as whl_name (mirroring pip.whl_mods) that is a simplified name that would apply to all downloads of this wheel.

Thoughts?

Activity

  1. aignas commented on Oct 7, 2024

    @aignas
    Collaborator

    It could be a bug, so without a repro, it is hard to say. I have the same expectations as you.

  2. keith commented on Oct 11, 2024

    @keith
    MemberAuthor

    You can tell it's patching both because even though I only have this in the MODULE.bazel:

    pip.override(
        file = "torch-2.4.0+cu121-cp311-cp311-linux_x86_64.whl",
        patch_strip = 1,
        patches = [
            "//:gpu-torch.patch",
            "//:gpu-torch-record.patch",
        ],
    )

    I get this warning:

    --- a/torch-2.3.0+cpu.dist-info/RECORD
    +++ b/torch-2.3.0+cpu.dist-info/RECORD
    @@ -10496,13 +10496,6 @@
     torch/xpu/_utils.py,sha256=JTETOkI3J7tp4SFKV3ukRlvQDO-6Qg850KkJ8wO6gh0,1591
     torch/xpu/random.py,sha256=gIwDBj51xPEfwa4a_xH-1oyTJO9AQdBHPXBC-PgpmEs,5217
     torch/xpu/streams.py,sha256=RWI7Y1ARIOAQfIiYL9y2qC4avqd9oPsuK7uVIefUI3E,5470
    -torch-2.3.0+cpu.dist-info/LICENSE,sha256=nCkW1Dsriv9qPGZi7VhLay19bc-E3XJxgd7q3FRyIU4,351851
    -torch-2.3.0+cpu.dist-info/METADATA,sha256=UKror7Nus0UqDlw1jlcQpwxmxS32jLzYrGJeMm5KQk8,24786
    -torch-2.3.0+cpu.dist-info/NOTICE,sha256=wsx78MrsdlLCtGCopHC-oWd_JB5KuOQx3zTPF_Wp_sA,23632
    -torch-2.3.0+cpu.dist-info/WHEEL,sha256=heXqORHgAzVHsSEcHHvvTohM_YB7MmN0HUQQME76CM0,105
    -torch-2.3.0+cpu.dist-info/entry_points.txt,sha256=SRhyGhohzXtwg-GPZHrgubQLqk1v5i-kOQiV-mj5fms,296
    -torch-2.3.0+cpu.dist-info/top_level.txt,sha256=MsBcfJyMU15lW1efu5w7Tzd4MenrYHiuaixbHMfAoco,25
    -torch-2.3.0+cpu.dist-info/RECORD,,
     torchgen/__init__.py,sha256=iirTpG38WcCsNMhEbi1dg7_jad6ptk_uzZ-BzaGBFyU,348
     torchgen/code_template.py,sha256=z3N3FvXHfvO2aLIu2LoFqe7XGpzbFfzXEyzhGez2KME,2903
     torchgen/context.py,sha256=798e45g0zoR69Xn_4HjTuBRXbuNnyyK-j5_vWOnrs_s,3974
    @@ -10642,3 +10635,10 @@
     torchgen/static_runtime/config.py,sha256=fEMB4EdO8aX47aW13s-nVSy-yM5qIzfXrOYaQkmv3-A,14493
     torchgen/static_runtime/gen_static_runtime_ops.py,sha256=Esr32XW78YkLpGAGhIw6ZsxmkwgI_Pd3j0rh585ijow,7347
     torchgen/static_runtime/generator.py,sha256=Sfe8TR6inZv-FHhJAG6gXBFXutPVMO8I4Uzw3xnGdd4,26374
    +torch-2.3.0+cpu.dist-info/LICENSE,sha256=nCkW1Dsriv9qPGZi7VhLay19bc-E3XJxgd7q3FRyIU4,351851
    +torch-2.3.0+cpu.dist-info/METADATA,sha256=UKror7Nus0UqDlw1jlcQpwxmxS32jLzYrGJeMm5KQk8,24786
    +torch-2.3.0+cpu.dist-info/NOTICE,sha256=wsx78MrsdlLCtGCopHC-oWd_JB5KuOQx3zTPF_Wp_sA,23632
    +torch-2.3.0+cpu.dist-info/WHEEL,sha256=heXqORHgAzVHsSEcHHvvTohM_YB7MmN0HUQQME76CM0,105
    +torch-2.3.0+cpu.dist-info/entry_points.txt,sha256=SRhyGhohzXtwg-GPZHrgubQLqk1v5i-kOQiV-mj5fms,296
    +torch-2.3.0+cpu.dist-info/top_level.txt,sha256=MsBcfJyMU15lW1efu5w7Tzd4MenrYHiuaixbHMfAoco,25
    +torch-2.3.0+cpu.dist-info/RECORD,,
    

    where you can see it's patching the 2.3.0 version, even though the wheel name I used includes 2.4.0. It's actually really lucky that the other patch I have even applies to multiple versions.

    In this case the patching actually takes a ton of time too, since this wheel is pretty large, so that's an extra downside.

    To repro in this project: repro.zip

    Run bazel build ..., probably has to be on a linux x86_64 box.

  3. keith commented on Oct 11, 2024

    @keith
    MemberAuthor

    If right here https://github.com/bazelbuild/rules_python/blob/71417c5ab792948ef87b3ce5bde0fb92be4aad83/python/private/pypi/extension.bzl#L431

    I add print(attr.file, whl_name) you can see it gets transformed from torch-2.4.0+cu121-cp311-cp311-linux_x86_64.whl to torch

  4. added a commit that references this issue on Oct 11, 2024
  5. keith commented on Oct 11, 2024

    @keith
    MemberAuthor

    here's the fix: #2294

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions