Skip to content

Pip install uses wrong Macos SDK version from standalone interpreters. #646

Closed

Description

@twheys

Hi, my team discovered an issue trying to use pip_install to install python dependencies that use cpython on an M1 Mac with a full xcode installation.

Here's the error from the build logs:

  building '_ruamel_yaml' extension
  creating build/temp.macosx-11.0-arm64-3.9
  Compiling with an SDK that doesn't seem to exist: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk
  Please check your Xcode installation
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -fPIC -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpjbwqfg64/tools/deps/include -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpjbwqfg64/tools/deps/include/ncursesw -I/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpjbwqfg64/tools/deps/include/uuid -Werror=unguarded-availability-new -g0 -I/private/var/tmp/_bazel_alex/bfcdc3c2b5b01e8ec0b8b8dd179fe4b8/external/r9python_darwin_arm64/include/python3.9 -c _ruamel_yaml.c -o build/temp.macosx-11.0-arm64-3.9/_ruamel_yaml.o
  clang: warning: no such sysroot directory: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk' [-Wmissing-sysroot]
  In file included from _ruamel_yaml.c:6:
  /private/var/tmp/_bazel_alex/bfcdc3c2b5b01e8ec0b8b8dd179fe4b8/external/r9python_darwin_arm64/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
  #include <stdio.h>
           ^~~~~~~~~

Running pip install on the same requirements outside of Bazel works as expected. The issue appears to be that it's looking for
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk
although the machine has a MacOS SDK version 12.1.

I'm not really sure how it's landing on SDK v11.3 but as a workaround, we've been manually installing MacOSX11.3.sdk at that file path.

Any help debugging this issue would be appreciated.

Here's some additional debug info

$ gcc -E -xc++ - -v
Apple clang version 13.0.0 (clang-1300.0.29.30)
Target: x86_64-apple-darwin21.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-strict-return -fno-rounding-math -munwind-tables -target-sdk-version=12.1 -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -target-linker-version 711 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/usr/local/include -stdlib=libc++ -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1 -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -fdeprecated-macro -fdebug-compilation-dir /Users/alex/r9/git -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fno-cxx-modules -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcommon -fcolor-diagnostics -clang-vendor-feature=+nullptrToBoolConversion -clang-vendor-feature=+messageToSelfInClassMethodIdReturnType -clang-vendor-feature=+disableInferNewAvailabilityFromInit -clang-vendor-feature=+disableNeonImmediateRangeCheck -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+revert09abecef7bbf -mllvm -disable-aligned-alloc-awareness=1 -mllvm -enable-dse-memoryssa=0 -o - -x c++ -
clang -cc1 version 13.0.0 (clang-1300.0.29.30) default target x86_64-apple-darwin21.3.0
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.

Activity

  1. caseyduquettesc commented on Mar 15, 2022

    @caseyduquettesc

    I bunch of libraries that have c-extensions use the values in your _sysconfigdata file in the stdlib. The compiler name and Xcode path used to build the interpreter are hardcoded in there and it doesn't look like the toolchain is sanitizing that so I'm guessing those wrong values are from the person who built python-build-standalone.

    Our workaround, which is not ideal, is a repo rule that replaces those paths with the result of xcode-select -p at runtime, so there's a startup cost.

  2. dhalperi commented on Mar 21, 2022

    @dhalperi
    Contributor

    Seems like the same issue on macOS 12.3 with Xcode CLT only, on Intel Mac. https://bazelbuild.slack.com/archives/CA306CEV6/p1647900906891929

  3. dhalperi commented on Mar 21, 2022

    @dhalperi
    Contributor

    It looks like the main issue is that the folks who ship these standalone interpreters don't support this:

    astral-sh/python-build-standalone#103 (comment)

    ... Neither of these approaches have been explored that much because most current users of these interpreters want a way to run Python applications and don't necessarily care about Python developer use cases.

  4. twheys commented on Apr 12, 2022

    @twheys
    Author

    @dhalperi could you please share the discussion from Slack? I don't know if I have access to the link you posted. I'd be interested in reading what was written there.

  5. dhalperi commented on Apr 12, 2022

    @dhalperi
    Contributor

    dhalperi Mar 21st at 3:15 PM
    Does the new rules_python support for standalone builds require Xcode to be installed as opposed to just the Xcode command-line-tools? (macOS latest - 12.3)
    I'm getting this when trying to build C libraries after switching to the new stuff:
    Compiling with an SDK that doesn't seem to exist: /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
    Please check your Xcode installation
    20 replies

    Jeremy Volkman 22 days ago
    this is because information from the platform that built these binaries is embedded within. E.g.,

    sysconfig.get_config_var('CONFIGURE_CFLAGS')
    '-arch x86_64 -mmacosx-version-min=10.9 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -fPIC -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp5g80mkyj/tools/deps/include -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp5g80mkyj/tools/deps/include/ncursesw -I/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmp5g80mkyj/tools/deps/include/uuid -Werror=unguarded-availability-new

    Jeremy Volkman 22 days ago
    not sure of a great workaround yet; people here that have been affected have just created symlinks

    dhalperi 22 days ago
    interesting, thanks

    dhalperi 22 days ago
    It looks like the main issue is that the folks who ship these standalone interpreters don't support this:
    astral-sh/python-build-standalone#103 (comment)
    ... Neither of these approaches have been explored that much because most current users of these interpreters want a way to run Python applications and don't necessarily care about Python developer use cases.

    Jeremy Volkman 22 days ago
    maybe indygreg will be open to suggestions now given https://twitter.com/indygreg/status/1503955727559245824

    matt 22 days ago
    Can set CFLAGS etc with flags like -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk etc to override these

    dhalperi 22 days ago
    Where in WORKSPACE (e.g.,) does that go? In a user-independent way 😉

    dhalperi 22 days ago
    I guess we have been "lucky" by our current terrible method to build interpreter locally - no caching, but this same problem is handled cleanly.

    Jeremy Volkman 22 days ago
    I'm working on some rules to pull package compilation out of WORKSPACE time and into build actions. No idea if they'll work but it's my hopeful resolution for folks at my org

    matt 22 days ago
    CFLAGS is an env var, so set it on whatever rule is running the pip install.

    dhalperi 22 days ago
    So the idea I guess would be to find the xcode_locator tool, use that to produce cflags, and pass that into the rules_python somewhere

    Casey Duquette 22 days ago
    fix_sysconfig.py

    Casey Duquette 22 days ago
    We have this really nasty hack that packages the interpreter with a "fixed" sysconfig file. Fwiw I'll paste it here, but I would definitely appreciate seeing it fixed properly. Making a portable python is tricky. (edited)

    Jeremy Volkman 22 days ago
    Python packaging is such a mess
    👍:skin-tone-2:👍
    4
    💯
    1

    Jeremy Volkman 12 days ago
    Just learned about the _PYTHON_SYSCONFIGDATA_NAME env var which provides a way to override sysconfig with some other module. Is this what you use @Casey Duquette?

    Casey Duquette 12 days ago
    We shamelessly overwrite the default in a repo rule

    Casey Duquette 12 days ago
    One reason was because we used clang to build the interpreter, but we don't have clang on every CI machine so we have to switch the compiler defined in the sysconfigdata to gcc. And then the xcode location could be different in CI where we have several versions installed so it has to pick the active one.

    Jeremy Volkman 12 days ago
    do you do any cross compilation?

    Jeremy Volkman 12 days ago
    Oh, I just realized you shared your code previously. Missed that

    Casey Duquette 12 days ago
    No, haven't attempted to get that working

  6. dhalperi commented on Apr 12, 2022

    @dhalperi
    Contributor

    the referenced fix_sysconfig.py file:

    # Where {FILE} is sysconfig_path
    exec(open('{FILE}').read())
    # replace sysroot with symlink
    for k,v in build_time_vars.items():
        if isinstance(v, str) and "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk" in v:
            build_time_vars[k] = v.replace("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk", "{xcode_root}/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk")
        if isinstance(v, str) and "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk" in v:
            build_time_vars[k] = v.replace("/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk", "{xcode_root}/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk")
    # update file
    f = open('{FILE}', 'w+')
    # get xcode path (will have a startup penalty)
    f.write("import subprocess\\n")
    f.write("xcode_root=subprocess.run(['xcode-select', '-p'], capture_output=True).stdout.strip()\\n")
    f.write("build_time_vars = " + repr(build_time_vars) + "\\n")
    # dynamically insert correct sysroot at runtime (will have another startup penalty)
    f.write("for k,v in build_time_vars.items():\\n")
    f.write("    if isinstance(v, str) and '{xcode_root}' in v:\\n")
    f.write("        build_time_vars[k] = v.replace('{xcode_root}', xcode_root.decode('utf-8') if xcode_root else '/Applications/Xcode.app/Contents/Developer')\\n")
    f.close()
  7. dhalperi commented on Apr 12, 2022

    @dhalperi
    Contributor

    FWIW, I think that anyone can join that slack channel. May need to create an account with email, not sso, however

  8. self-assigned this
    on May 8, 2022
  9. changed the title [-]Pip install uses wrong Macos SDK version on M1 mac with full xcode installation[/-] [+]Pip install uses wrong Macos SDK version from standalone interpreters.[/+] on May 10, 2022
  10. tuananh commented on Jun 9, 2022

    @tuananh

    i have the same issue. anyone find a workaround for this yet?

  11. hrfuller commented on Jul 1, 2022

    @hrfuller
    Contributor

    Closed via #697

  12. Mercanuis commented on Jul 27, 2022

    @Mercanuis

    I think this should be reopened, as this error seems to still be occurring.

    Some details
    Bazel Version: 5.2.0
    Xcode Version: via

     ~/ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables | grep version
    version: 13.0.0.0.1.1627064638
    
     ~/ gcc -E -xc++ - -v                                                  
    
    Apple clang version 13.1.6 (clang-1316.0.21.2.5)
    Target: arm64-apple-darwin21.6.0
    Thread model: posix
    InstalledDir: /Library/Developer/CommandLineTools/usr/bin
     "/Library/Developer/CommandLineTools/usr/bin/clang" -cc1 -triple arm64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -Werror=implicit-function-declaration -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mframe-pointer=non-leaf -fno-strict-return -fno-rounding-math -munwind-tables -target-sdk-version=12.3 -fvisibility-inlines-hidden-static-local-var -target-cpu apple-m1 -target-feature +v8.5a -target-feature +fp-armv8 -target-feature +neon -target-feature +crc -target-feature +crypto -target-feature +dotprod -target-feature +fp16fml -target-feature +ras -target-feature +lse -target-feature +rdm -target-feature +rcpc -target-feature +zcm -target-feature +zcz -target-feature +fullfp16 -target-feature +sm4 -target-feature +sha3 -target-feature +sha2 -target-feature +aes -target-abi darwinpcs -fallow-half-arguments-and-returns -debugger-tuning=lldb -target-linker-version 763 -v -resource-dir /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -I/usr/local/include -stdlib=libc++ -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1 -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -internal-externc-isystem /Library/Developer/CommandLineTools/usr/include -Wno-reorder-init-list -Wno-implicit-int-float-conversion -Wno-c99-designator -Wno-final-dtor-non-final-class -Wno-extra-semi-stmt -Wno-misleading-indentation -Wno-quoted-include-in-framework-header -Wno-implicit-fallthrough -Wno-enum-enum-conversion -Wno-enum-float-conversion -Wno-elaborated-enum-base -Wno-reserved-identifier -Wno-gnu-folding-constant -Wno-objc-load-method -fdeprecated-macro -fdebug-compilation-dir=/Users/joseph.orme/Workspaces/cityos -ferror-limit 19 -stack-protector 1 -fstack-check -mdarwin-stkchk-strong-link -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fno-cxx-modules -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcommon -fcolor-diagnostics -clang-vendor-feature=+messageToSelfInClassMethodIdReturnType -clang-vendor-feature=+disableInferNewAvailabilityFromInit -clang-vendor-feature=+disableNonDependentMemberExprInCurrentInstantiation -fno-odr-hash-protocols -clang-vendor-feature=+enableAggressiveVLAFolding -clang-vendor-feature=+revert09abecef7bbf -clang-vendor-feature=+thisNoAlignAttr -clang-vendor-feature=+thisNoNullAttr -mllvm -disable-aligned-alloc-awareness=1 -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o - -x c++ -
    clang -cc1 version 13.1.6 (clang-1316.0.21.2.5) default target arm64-apple-darwin21.6.0
    ignoring nonexistent directory "/usr/local/include"
    ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
    ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
    #include "..." search starts here:
    #include <...> search starts here:
     /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1
     /Library/Developer/CommandLineTools/usr/lib/clang/13.1.6/include
     /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
     /Library/Developer/CommandLineTools/usr/include
     /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
    End of search list.
    

    Computer: Mac M1 Pro Max

  13. Mercanuis commented on Jul 27, 2022

    @Mercanuis

    Sorry for the panic comment on this thread. It turns out that the rules_python that my project is using is 0.8.1 and this fix seems to be only in newer versions, an update to the rules 0.10.2 in my WORKSPACE seems to resolve my issue. Sorry for any problems I might have caused!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions