Repository navigation
Description
🐞 bug report
Affected Rule
The issue is caused by the rule:py_binary rule
Is this a regression?
Yes. Well behaved code loading everything from rules_python (as recommended and desired) stopped working.
A workaround is possible if you control the rule that consumes the target that should provide PyRuntimeInfo: accept both the rules_python PyRuntimeInfo symbol and builtin PyRuntimeInfo provider. When accessing it in rule code, prefer the rules_python one over the bulitin one. When forwarding, forward both if you need compatibility. The gist of the code would look like this:
load("@rules_python//python:py_runtime_info.bzl", RulesPythonPyRuntimeInfo = "PyRuntimeInfo")
BuiltinPyRuntimeInfo = PyRuntimeInfo
foo_binary = rule(..., attrs = {
# Note its a 2-element list of 1-element lists.
"bin": attr.label(provides = [[RulesPythonPyRuntimeInfo], [BuiltinPyRuntimeInfo]])
})
def foo_binary_impl(ctx):
bin = ctx.attr.bin
if RulesPythonPyRuntimeInfo in bin:
rp_info = bin[RulesPythonPyRuntimeInfo]
providers.append(rp_info)
else:
rp_info = None
if BuiltinPyRuntimeInfo in bin:
builtin_info = bin[BuiltinPyRuntimeInfo]
providers.append(builtin_info)
else:
builtin_info = None
return providers
Description
Using py_binary rule triggers fail() called with error string "target {} does not have rules_python PyRuntimeInfo or builtin PyRuntimeInfo)" on Windows 10
🔬 Minimal Reproduction
Load PyRuntimeInfo and py_binary from rules_python, then try to read the loaded PyRuntimeInfo from the py_binary target.
# BUILD
load(":defs.bzl", "foo")
load("@rules_python//python:py_binary.bzl", "py_binary")
foo(name="foo", bin=":bin")
py_binary(name="bin", srcs=["bin.py"])
# defs.bzl
load("@rules_python//python:py_runtime_info.bzl", "PyRuntimeInfo")
foo = rule(attrs = {
"bin": attr.label(provides=[PyRuntimeInfo])
}, ...
)
🔥 Exception or Error
An error about PyRuntimeInfo not being in the target being depended upon.
When it occurs through provided =..., it will fail the build pretty early with a Bazel-generated error
If you have rule code doing target[PyRuntimeInfo], it'll be an error pointing to the line in your rule code looking up that provider.
If the version-aware rules are used, it might result in an error like the below:
bazel run @hedron_compile_commands//:refresh_all
ERROR: C:/users/marki/_bazel_marki/alqgzwqn/external/hedron_compile_commands/BUILD:7:25: in _transition_py_binary rule @@hedron_compile_commands//:refresh_all:
Traceback (most recent call last):
File "C:/users/marki/_bazel_marki/alqgzwqn/external/rules_python/python/config_settings/transition.bzl", line 78, column 13, in _transition_py_impl
fail("target {} does not have rules_python PyRuntimeInfo or builtin PyRuntimeInfo".format(target))
Error in fail: target does not have rules_python PyRuntimeInfo or builtin PyRuntimeInfo
ERROR: C:/users/marki/_bazel_marki/alqgzwqn/external/hedron_compile_commands/BUILD:7:25: Analysis of target '@@hedron_compile_commands//:refresh_all' failed
ERROR: Analysis of target '@@hedron_compile_commands//:refresh_all' failed; build aborted
🌍 Your Environment
Output of bazel version:
bazel 7.0.2
Rules_python version:
0.29.0 for sure. Likely 0.28.x, too, but I didn't check.
Anything else relevant?
Related issue in Hedron's compile-commands-extractor project hedronvision/bazel-compile-commands-extractor#163
Activity
axbycc-mark commentedon Jan 30, 2024
cc @cpsauer
rickeylev commentedon Jan 31, 2024
I'm somewhat skeptical that this is an issue in rules_python, or at the least is a new issue in a recent version. The logic producing that particular failure message was introduced in Dec 2023 in #1604, however, the prior logic was equivalent (it also required the PyRuntimeInfo provider because it did
target[PyRuntimeInfo]), and has been around since Nov 2022 in #846.The error message is also odd: the target name is blank. I don't know how that would happen, but is certainly very strange.
cpsauer commentedon Jan 31, 2024
Hey, thanks for replying back so fast Richard.
Context here is that I'd recently moved a (fairly popular) bazel tool we wrote over to rules_python to get hermetic python and pip_parse, but there have been some issues like this one that mean we might have to revert. I'm moderately confident that it's a rules_python (though of course, possibly upstream, bazel core) issue, since this works fine on other platforms (macOS and linux). @axbycc-mark has confirmed the issue with both rules_python 0.28 and 0.29, but that all works fine before our switch to rules_python.
Good spot on the missing target name. Is it possible rules_python breaks on Windows if py_binary is called from a macro, as it is here, causing the target to not get picked up somehow? Fairly simple wrapper-macro here otherwise. Would love your help scoping what might be wrong here.
I presume we're pretty sure that py_binary works more generally on Windows? @axbycc-mark, to narrow things down, I'll push up a commit adding a non-macro py_binary to test. If you could grab the latest commit of the tool (backlink incoming) and then see if
bazel build @hedron_compile_commands//:nvcc_clang_diffsucceeds or fails the same way, that'd be awesome.Thanks,
Chris
(ex-Googler)
Test case for
rickeylev commentedon Jan 31, 2024
Can you try a 0.27.0 or earlier? The 0.28.0 release contains some changes to py_runtime and PyRuntimeInfo, however they should be transparent.
re: empty target name: When I say weird, I mean it's an impossible code path, as in, the setting of that attribute involves a non-empty string literal. See https://github.com/bazelbuild/rules_python/blob/main/python/config_settings/transition.bzl#L250
So, worst case, it would have the value "_".
The only thing I can think of is a strange interaction when mixing the bazel-bundled py_runtime rule and/or PyRuntimeInfo objects and the rule_python ones. In case it wasn't clear, there are "two" PyRuntimeInfo objects -- the ones defined by rules_python, and the ones defined by Bazel itself. However, the particular code that is failing is compatibility code that is looking for either of those PyRuntimeInfo objects.
Hm, maybe a n interaction between workspace and bzlmod? In Bazel 7, bzlmod is enabled by default, but stuff in WORKSPACE can bleed over into bzlmod. Do things work if you set
--noenable_bzlmod? Alternatively, create an empty file namedWORKSPACE.bzlmod(along side WORKSPACE); this will prevent WORKSPACE from bleeding over into bzlmod. You can also try building with Bazel 6.4, where bzlmod is disabled by default.Also, in case it wasn't clear, loading the defs from the versioned repos means they are being bound to that specific python version. i.e.
load("@python_3_11//:defs.bzl", "py_binary")means it's going to use a py_binary that forces itself to find a 3.11 compatible interpreter. Using these is more common for build tools (or other special situations) when you don't want the python version to change out from under you. Otherwise, load the version-unaware rules,load("@rules_python//python:py_binary.bzl").cpsauer commentedon Feb 1, 2024
@axbycc-mark has the advantage on testing, so I'll defer to him, but it certainly seems like it's hitting that case somehow! Definitely seems worth running the test with --noenable_bzlmod
Richard, do you guys have the ability to test on windows and investigate?
Re the python version: Yep, got it & intentional. This is a build/dev tool seeking reproducible hermetic python for its implementation, rather than exposing to users.
Switch back off rules_python.
axbycc-mark commentedon Feb 1, 2024
Here are my results with the --noenable_bzlmod command from the reproduction repo.
cpsauer commentedon Feb 1, 2024
^ Oh, interesting! Looks like it's a windows-only rules_python bzlmod interaction issue, then!
Great hypothesis @rickeylev, and thanks for testing @axbycc-mark.
@rickeylev, any idea what might be going on?
[That other error is part of the tool, not rules_python, already fixed in a subsequent commit; rules_python is working as expected with --noenable_bzlmod.]
cpsauer commentedon Feb 1, 2024
@axbycc-mark, would you be down to still point to hedronvision/bazel-compile-commands-extractor@40a51d6 and try
bazel build @hedron_compile_commands//:nvcc_clang_diffwith and without --noenable_bzlmod?rickeylev commentedon Feb 1, 2024
That looks like an error in the refresh_all.py code. removeprefix is a method of python str objects, but it sounds like it has a WindowsPath object from pathlib. It's a fairly easy mistake to make. A function accepting a string path often doesn't notice because it just passes the value onto a lower-level API that understands Path objects.
cpsauer commentedon Feb 2, 2024
^yep, as above! Looks like you were right that the original issue is a windows-specific rules_python bazelmod interaction?
axbycc-mark commentedon Feb 2, 2024
@cpsauer
bzlmod
no bzlmod
cpsauer commentedon Feb 2, 2024
Thanks for testing! That helps narrow things down: Looks like this has nothing to do with macros or other fancy uses of py_binary. Instead, all is consistent with Richard's hypothesis that rules_python is clashing with itself when brought in via the workspace in Bazel 7 on Windows when bzlmod defaults to being on. Perhaps because a rules_python version is brought in by default as a Bazel tool?
For now, ofc, please hop back to the latest commit of the tool where all should work (we've temporarily reverted our rules_python usage until this and some other issues are fixed). Thanks for all your help tracking this down!
aignas commentedon Feb 3, 2024
@cpsauer, if you want to things not clash, you should check out WORKSPACE.bzlmod usage.
cpsauer commentedon Feb 9, 2024
^ indeed, but folks are going to use the tool and rules_python from the WORKSPACE for a good while--isn't that a case worth supporting?
rickeylev commentedon Feb 9, 2024
I'm very sure the bug this is talking about is the same one @mattem was able to show a simpler repro for last week, so I'll repurpose this to that. I don't fully understand how these hadron rules are triggering it, but the cause would be the same.
The basic issue is the switch to using the rules_python PyRuntimeInfo object overlooked that some users do want to get the PyRuntimeInfo object out of e.g. py_binary. This created the situation where the builtin py_binary rules provide the builtin PyRuntimeInfo object, but the rules_python PyRuntimeInfo binding was pointing to the rules_python PyRuntimeInfo definition. Because providers operate by identity, the two aren't considered the same.
I'll switch the PyRuntimeInfo symbol back to the builtin one for now. Instead it'll be guarded by pystar being enabled
I have a fix with tests I'm about ready to send for review. It'll go into the next release, planned for once this fix is merged.
[-]Windows 10: rules_python complaining about it's own py_binary rule[/-][+]bug: PyRuntimeInfo provided by py_binary doesn't match PyRuntimeInfo symbol from rules_python[/+]cpsauer commentedon Feb 10, 2024
Thanks so much for tackling, @rickeylev!
cpsauer commentedon Feb 10, 2024
@axbycc-mark, could you give that PR's commit a quick test from your workspace, just to make sure that fixes things? Just spin up a quick py_binary and make sure all works (but didn't on the prior release).
Thanks so much all 💛
fix(PyRuntimeInfo): use builtin PyRuntimeInfo unless pystar is enable…