Repository navigation
Closed
Description
Hi, my team discovered an issue trying to use pip_install to install python dependencies that use cpython on an M1 Mac with a full xcode installation.
Here's the error from the build logs:
building '_ruamel_yaml' extension
creating build/temp.macosx-11.0-arm64-3.9
Compiling with an SDK that doesn't seem to exist: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk
Please check your Xcode installation
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -fPIC -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpjbwqfg64/tools/deps/include -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpjbwqfg64/tools/deps/include/ncursesw -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpjbwqfg64/tools/deps/include/uuid -Werror=unguarded-availability-new -g0 -I/private/var/tmp/_bazel_alex/bfcdc3c2b5b01e8ec0b8b8dd179fe4b8/external/r9python_darwin_arm64/include/python3.9 -c _ruamel_yaml.c -o build/temp.macosx-11.0-arm64-3.9/_ruamel_yaml.o
clang: warning: no such sysroot directory: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk' [-Wmissing-sysroot]
In file included from _ruamel_yaml.c:6:
/private/var/tmp/_bazel_alex/bfcdc3c2b5b01e8ec0b8b8dd179fe4b8/external/r9python_darwin_arm64/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
^~~~~~~~~
Running pip install on the same requirements outside of Bazel works as expected. The issue appears to be that it's looking for
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk
although the machine has a MacOS SDK version 12.1.
I'm not really sure how it's landing on SDK v11.3 but as a workaround, we've been manually installing MacOSX11.3.sdk at that file path.
Any help debugging this issue would be appreciated.
Here's some additional debug info
$ gcc -E -xc++ - -v
Apple clang version 13.0.0 (clang-1300.0.29.30)
Target: x86_64-apple-darwin21.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -fno-rounding-math -munwind-tables -target-sdk-version=12.1 -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -target-linker-version 711 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/usr/local/include -stdlib=libc++ -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1 -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -fdeprecated-macro -fdebug-compilation-dir /Users/alex/r9/git -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fno-cxx-modules -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcommon -fcolor-diagnostics -clang-vendor-feature=+nullptrToBoolConversion -clang-vendor-feature=+messageToSelfInClassMethodIdReturnType -clang-vendor-feature=+disableInferNewAvailabilityFromInit -clang-vendor-feature=+disableNeonImmediateRangeCheck -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+revert09abecef7bbf -mllvm -disable-aligned-alloc-awareness=1 -mllvm -enable-dse-memoryssa=0 -o - -x c++ -
clang -cc1 version 13.0.0 (clang-1300.0.29.30) default target x86_64-apple-darwin21.3.0
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
Reacted by Rohan Agarwal and hao-affirm
Metadata
Metadata
Assignees
Labels
No labels
Activity
caseyduquettesc commentedon Mar 15, 2022
I bunch of libraries that have c-extensions use the values in your
_sysconfigdatafile in the stdlib. The compiler name and Xcode path used to build the interpreter are hardcoded in there and it doesn't look like the toolchain is sanitizing that so I'm guessing those wrong values are from the person who built python-build-standalone.Our workaround, which is not ideal, is a repo rule that replaces those paths with the result of
xcode-select -pat runtime, so there's a startup cost.dhalperi commentedon Mar 21, 2022
Seems like the same issue on macOS 12.3 with Xcode CLT only, on Intel Mac. https://bazelbuild.slack.com/archives/CA306CEV6/p1647900906891929
dhalperi commentedon Mar 21, 2022
It looks like the main issue is that the folks who ship these standalone interpreters don't support this:
astral-sh/python-build-standalone#103 (comment)
twheys commentedon Apr 12, 2022
@dhalperi could you please share the discussion from Slack? I don't know if I have access to the link you posted. I'd be interested in reading what was written there.
dhalperi commentedon Apr 12, 2022
dhalperi Mar 21st at 3:15 PM
Does the new rules_python support for standalone builds require Xcode to be installed as opposed to just the Xcode command-line-tools? (macOS latest - 12.3)
I'm getting this when trying to build C libraries after switching to the new stuff:
Compiling with an SDK that doesn't seem to exist: /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
Please check your Xcode installation
20 replies
Jeremy Volkman 22 days ago
this is because information from the platform that built these binaries is embedded within. E.g.,
Jeremy Volkman 22 days ago
not sure of a great workaround yet; people here that have been affected have just created symlinks
dhalperi 22 days ago
interesting, thanks
dhalperi 22 days ago
It looks like the main issue is that the folks who ship these standalone interpreters don't support this:
astral-sh/python-build-standalone#103 (comment)
... Neither of these approaches have been explored that much because most current users of these interpreters want a way to run Python applications and don't necessarily care about Python developer use cases.
Jeremy Volkman 22 days ago
maybe indygreg will be open to suggestions now given https://twitter.com/indygreg/status/1503955727559245824
matt 22 days ago
Can set CFLAGS etc with flags like -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk etc to override these
dhalperi 22 days ago
Where in WORKSPACE (e.g.,) does that go? In a user-independent way 😉
dhalperi 22 days ago
I guess we have been "lucky" by our current terrible method to build interpreter locally - no caching, but this same problem is handled cleanly.
Jeremy Volkman 22 days ago
I'm working on some rules to pull package compilation out of WORKSPACE time and into build actions. No idea if they'll work but it's my hopeful resolution for folks at my org
matt 22 days ago
CFLAGS is an env var, so set it on whatever rule is running the pip install.
dhalperi 22 days ago
So the idea I guess would be to find the xcode_locator tool, use that to produce cflags, and pass that into the rules_python somewhere
Casey Duquette 22 days ago
fix_sysconfig.py
Casey Duquette 22 days ago
We have this really nasty hack that packages the interpreter with a "fixed" sysconfig file. Fwiw I'll paste it here, but I would definitely appreciate seeing it fixed properly. Making a portable python is tricky. (edited)
Jeremy Volkman 22 days ago
Python packaging is such a mess
👍:skin-tone-2:👍
4
💯
1
Jeremy Volkman 12 days ago
Just learned about the _PYTHON_SYSCONFIGDATA_NAME env var which provides a way to override sysconfig with some other module. Is this what you use @Casey Duquette?
Casey Duquette 12 days ago
We shamelessly overwrite the default in a repo rule
Casey Duquette 12 days ago
One reason was because we used clang to build the interpreter, but we don't have clang on every CI machine so we have to switch the compiler defined in the sysconfigdata to gcc. And then the xcode location could be different in CI where we have several versions installed so it has to pick the active one.
Jeremy Volkman 12 days ago
do you do any cross compilation?
Jeremy Volkman 12 days ago
Oh, I just realized you shared your code previously. Missed that
Casey Duquette 12 days ago
No, haven't attempted to get that working
dhalperi commentedon Apr 12, 2022
the referenced
fix_sysconfig.pyfile:dhalperi commentedon Apr 12, 2022
FWIW, I think that anyone can join that slack channel. May need to create an account with email, not sso, however
[-]Pip install uses wrong Macos SDK version on M1 mac with full xcode installation[/-][+]Pip install uses wrong Macos SDK version from standalone interpreters.[/+]tuananh commentedon Jun 9, 2022
i have the same issue. anyone find a workaround for this yet?
hrfuller commentedon Jul 1, 2022
Closed via #697
Mercanuis commentedon Jul 27, 2022
I think this should be reopened, as this error seems to still be occurring.
Some details
Bazel Version: 5.2.0
Xcode Version: via
Computer: Mac M1 Pro Max
Mercanuis commentedon Jul 27, 2022
Sorry for the panic comment on this thread. It turns out that the
rules_pythonthat my project is using is 0.8.1 and this fix seems to be only in newer versions, an update to the rules 0.10.2 in my WORKSPACE seems to resolve my issue. Sorry for any problems I might have caused!bazel: update grpcio-tools to fix mac build on arm
bazel: update grpcio-tools to fix mac build on arm
bazel: update grpcio-tools to fix mac build on arm
bazel: update grpcio-tools to fix mac build on arm
bazel: update grpcio-tools to fix mac build on arm
bazel: update grpcio-tools to fix mac build on arm