Repository navigation
Description
🐞 bug report
Affected Rule
pip.parse
Is this a regression?
I don't think so. Nothing on the Bazel or rules_python side of things have changed when this appeared.
The bug appeared when our internal foobar package was updated from 2.2.10 to 2.2.12.
Description
pip.parse is installing a Py3.12 version of a (private-index) wheel even though we're using Py3.11.
How does rules_python figure out which Python wheel to install if there are multiple wheels for different python versions?
We have an package on our internal private index. It has wheels for two python versions:
foobar-2.2.12-py311-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
foobar-2.2.12-py312-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
In MODULE.bazel, I set python_version=3.11:
python.toolchain(
configure_coverage_tool = True,
python_version = "3.11",
)
...
pip.parse(
experimental_extra_index_urls = ["https://[redacted]/simple"],
experimental_index_url = "https://pypi.org/simple",
hub_name = "pypi",
python_version = "3.11",
requirements_lock = "//:requirements.txt",
)However, the Py3.12 wheel gets installed:
$ cat bazel-bin/.../decoding_test.runfiles/rules_python~~pip~pypi_311_foobar/site-packages/foobar-2.2.12.dist-info/WHEEL
Wheel-Version: 1.0
Generator: bazel-wheelmaker 1.0
Root-Is-Purelib: true
Tag: py312-none-manylinux_2_17_x86_64
Tag: py312-none-manylinux2014_x86_64
Root-Is-Purelib: False
My requirements.txt has:
foobar==2.2.12 \
--hash=sha256:b7d2f441650f075a91f80d131673791f2b7face5722105f528916226dbd8e9e4 \
--hash=sha256:f8eac44fb8f2ca05d2335085651d5471f5f87647fbf1159b566923b7524eff55
- The first hash
b7d2...e9e4is for the 3.11 wheel, as verified bypython -m pip hash ./foobar-2.2.12-py311-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - The 2nd hash
f8ea...ff55is for the 3.12 wheel, as verified bypython -m pip hash ./foobar-2.2.12-py312-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
🔬 Minimal Reproduction
Working on this. Need to find a public pypi package that has separate 3.11 and 3.12 wheels. Know any?
🔥 Exception or Error
bazel build doesn't complain at all. During bazel test, Python throws this:
...
src/.../circuits_for_snake_circuit.py:9: in
import foobar
../rules_python~~pip~pypi_311_foobar/site-packages/foobar/__init__.py:4: in
from foobar import circuit_generation
E ImportError: Python version mismatch: module was compiled for Python 3.12, but the interpreter version is incompatible: 3.11.9 (main, Apr 15 2024, 18:24:23) [Clang 17.0.6 ].🌍 Your Environment
Operating System:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux rodete
Release: n/a
Codename: rodeteOutput of bazel version:
$ bazel version
Bazelisk version: v1.19.0
Starting local Bazel server and connecting to it...
Build label: 7.2.0rc1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Wed May 15 15:57:57 2024 (1715788677)
Build timestamp: 1715788677
Build timestamp as int: 1715788677Rules_python version:
bazel_dep(name = "bazel_skylib", version = "1.6.1")
bazel_dep(name = "rules_python", version = "0.31.0")
bazel_dep(name = "buildifier_prebuilt", version = "6.4.0", dev_dependency = True)
bazel_dep(name = "rules_python_gazelle_plugin", version = "0.31.0") # Same vers as rules_python
bazel_dep(name = "gazelle", version = "0.35.0", repo_name = "bazel_gazelle")
git_override(
module_name = "rules_python",
commit = "8791cbbaa2336e24555a6b577ed6e19df24d7d88",
remote = "https://github.com/bazelbuild/rules_python",
)
git_override(
module_name = "rules_python_gazelle_plugin",
commit = "8791cbbaa2336e24555a6b577ed6e19df24d7d88",
remote = "https://github.com/bazelbuild/rules_python",
strip_prefix = "gazelle",
)Anything else relevant?
This appeared when we updated foobar from 2.2.10 to 2.2.12.
For 2.2.10, the requirements.txt also had hashes for both Py3.11 and Py3.12.
Activity
aignas commentedon May 29, 2024
Yup, this is a know issue that would be fixed by #1917. Could you please try that one?
I would also expect the wheel to be cp312 rather than py312 because the version numbers are specific to an implementation I think. I wonder if our implementation should error out in those cases.
dougthor42 commentedon May 30, 2024
Oh sweet! I'll be able to test things tomorrow morning.
(Forgive me if you already know all of this - I'm writing it down mostly for my own sanity.)
TL;DR: I too would expect the wheel to be
cp312rather thanpy312. However, I don't know the full build stack of our internalfoobarpackage, so it's possible that it wasn't built withcpython. 🤷The wheel filename is defined by PEP 491 as
{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whland the various tags defined in PEP 425. The tags are implemented inwheel.bdist_wheel.bdist_wheel.get_tag(). Specifically, L343-L352 determine the "language implementation and version tag"{python tag}.For pure wheels (which
foobaris1, see thecat .../WHEELresults in OP), these lines callpython_tag()which simply returns the"py"plus the major version (eg:"py3"). Thebdist_wheelclass is a subclass of setuptools.Command, which itself is a subclass of thedistutils.cmd.CommandABC, both of which state that the values set byinitialize_options()may be overwritten during the build (such as by other Commands).It's possible that whatever build process we're using to make
foobarwheels ends up making such modifications to include the minor version"py3" --> "py312".For non-pure wheels, these lines end up calling functions from a vendored version of the
packagingpackage, specificallypackaging.tags.interpreter_name()andpackaging.tags.interpreter_version().interpreter_name()callssys.implementation.namewhich typically returns"cpython". This value is then looked up in thetags.INTERPRETER_SHORT_NAMESmapping and converted to the short-form"cp"....
Whoo, that was fun! I spent way too much time on that though, and while I learned a lot, I didn't find anything to say exactly why we're getting
py312instead ofcp312🤣.Error out in cases where the python tag is
pyinstead ofcp? Definitely not -pyXYZis still technically a valid tag.Or are you talking about something else?
Footnotes
At least I think it is. There are two conflicting
Root-Is-Purelib:lines so... 🤷 ↩fixes for bazel-contrib#1930
refactor/fix: store dists in parse_requirements output (#1917)