Skip to content

pip_install fails with error: command '/usr/bin/gcc' failed with exit code 1 #681

Closed

Description

@davians12

🐞 bug report

Affected Rule

The issue is caused by the rule: pip_install

Is this a regression?

Yes, the previous version in which this bug was not present was: Not a regression

Description

pip_install fails for only one requirement `tree-sitter` from `requirements.txt`

🔥 Exception or Error



Failed to build tree-sitter
 (  ERROR: Command errored out with exit status 1:
   command: /home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/python_bin /home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/pypi__pip/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpxxrp34vx
       cwd: /tmp/pip-wheel-hw8ksvfm/tree-sitter
  Complete output (20 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/tree_sitter
  copying tree_sitter/__init__.py -> build/lib.linux-x86_64-3.7/tree_sitter
  running build_ext
  building 'tree_sitter.binding' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/tree_sitter
  creating build/temp.linux-x86_64-3.7/tree_sitter/core
  creating build/temp.linux-x86_64-3.7/tree_sitter/core/lib
  creating build/temp.linux-x86_64-3.7/tree_sitter/core/lib/src
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g0 -fPIC -Itree_sitter/core/lib/include -Itree_sitter/core/lib/src -I/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/config-3.7m-x86_64-linux-gnu/Include -I/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter -c tree_sitter/binding.c -o build/temp.linux-x86_64-3.7/tree_sitter/binding.o -std=c99 -Wno-unused-variable
tree_sitter/binding.c:2:10: fatal error: Python.h: No such file or directory
   #include "Python.h"
            ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for tree-sitter
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/rules_python/python/pip_install/extract_wheels/__main__.py", line 5, in 
    main()
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/rules_python/python/pip_install/extract_wheels/__init__.py", line 87, in main
    subprocess.run(pip_args, check=True)
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/python_bin', '-m', 'pip', 'wheel', '-r', '/home/user/natural-cubert/requirements.txt']' returned non-zero exit status 1.
)

🌍 Your Environment

Operating System:

  
   Ubuntu
  

Output of bazel version:

  
Build label: 4.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jan 21 07:33:24 2021 (1611214404)
Build timestamp: 1611214404
  

Rules_python version:

  
0.1.0
  

Anything else relevant?
The bazel setup used to build and run, not sure what changed.

Activity

  1. hrfuller commented on May 8, 2022

    @hrfuller
    Contributor

    I believe the issue lies with the hermetic interpreters which are now used, apparently by default. The installed interpreters libs don't seem have the correct header files present.

  2. self-assigned this
    on May 8, 2022
  3. hrfuller commented on May 8, 2022

    @hrfuller
    Contributor

    @davians12 I can't repro this exact issue because I develop on macos. Can you put together a repro of the WORKSPACE which create this issue. In particular are you using a custom python interpreter toolchain target or the built in hermetic toolchains?

  4. removed their assignment
    on May 8, 2022
  5. thundergolfer commented on May 10, 2022

    @thundergolfer

    @hrfuller Yep this is totally expected with the python-build-standalone intepreter.

    pip_install fails for only one requirement tree-sitter from requirements.txt

    It's failed to find a wheel for tree-sitter so fell back to compiling it. It appears tree-sitter doesn't distribute a Linux wheel. https://pypi.org/project/tree-sitter/#files

  6. hrfuller commented on May 11, 2022

    @hrfuller
    Contributor

    So this issue is basically the linux equivalent of #646 but in the integration test I added to #679 I can't repro this failure with linux. To me that implies that the interpreter setup here is slightly different. In particular this include path -I/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/config-3.7m-x86_64-linux-gnu/Include doesn't look like it comes from the standalone interpreters because the external repo name in the path doesn't have the standard form that rules python creates e.g something like `python39_x86_64-apple-darwin. Thats why I wanted to see a repro.

  7. aaliddell commented on May 16, 2022

    @aaliddell
    Contributor

    I am seeing the same thing, using a custom Python toolchain and the hiredis and python-prctl packages. The build works with 0.7.0 and fails with 0.8.0, so I have bisected the issue back to fe30f15 where a set of dependencies were updated. It appears that new combination of packages has changed how the include paths are being found when creating the gcc compile args, resulting in the -I flags pointing to the wrong locations.

    I've tried looking through the issues on those packages for relevant changes, but not found anything obvious yet. The simple test would be to update these packages one-by-one or just bulk upgrade all of these to their latest version to see if there was a bug that's been resolved.

    This works (commit prior to fe30f15):

    http_archive(
        name = "rules_python",
        sha256 = "00360543865915c7492371b631e6824e1bd6976c88062242a65da0e6b18618c0",
        strip_prefix = "rules_python-0f189979467de61cb0512cc0a133d2122e041e90",
        url = "https://github.com/bazelbuild/rules_python/archive/0f189979467de61cb0512cc0a133d2122e041e90.zip",
    )

    This fails:

    http_archive(
        name = "rules_python",
        sha256 = "18a8d97f706d768169f93e251d02eea6f8e1810bd90c2fe78cfdf036aa1772ef",
        strip_prefix = "rules_python-fe30f1561e3afef59e3f8be6339c88444100de25",
        url = "https://github.com/bazelbuild/rules_python/archive/fe30f1561e3afef59e3f8be6339c88444100de25.zip",
    )

    @davians12: Do you see the same behaviour when you use the above rules_python versions?

    I'll try to extract a repro repo too.

  8. aaliddell commented on May 16, 2022

    @aaliddell
    Contributor

    So, this has been a rabbit hole into Python builds. The repo I have that reproduces this is similar in setup to the demo code you have here in this repo, where Python is pulled and built in patch commands. Given the .../bazel_install/... path segment shown in the top comment, I would guess @davians12 has the same setup, as this was a common-ish setup prior to the hermetic Python introduction recently. This has always been a bit of a hack and it seems we are finally seeing a consequence of that.

    The package update that breaks the build is setuptools between v60.1.1 and v60.2.0. Looking at the diff, there are changes to the way Python build config files are detected, which appears to have a knock-on effect in how the include paths are generated in build_ext when building the wheel.

    The section of interest appears to be in how distutils checks if a Python executable is part of a 'non-installed' build and changes behaviour if so. The critical piece here is the final Python binary is symlinked back into the source tree, which makes distutils think we are in an uninstalled Python build due to the presence of the source files alongside the binary.

    So, as far as I can tell, this is not a bug in rules_python nor any of the upstream Python packages, but rather a latent problem in mangling Python build directories that hasn't mattered until an unrelated cleanup in distutils exposed it.

    The fix for users with this build setup is to ensure that the final binary/symlink that is given to your toolchain/pip_install/pip_parse is not positioned alongside the source files, to prevent distutils getting confused. I have done this by separating the source and build dirs in the @python repo. Alternatively, switch to the new hermetic toolchains.

  9. groodt commented on May 17, 2022

    @groodt
    Contributor

    Coincidentally running into the same issue. The setuptools upgrade was introduced here: #661

  10. groodt commented on May 17, 2022

    @groodt
    Contributor

    I'm currently winding back setuptools versions to find out when it breaks in my environment. I'm not sure how necessary #661 was. I think we need to be more careful with setuptools upgrades and stay on old versions as long as possible.

  11. aaliddell commented on May 17, 2022

    @aaliddell
    Contributor

    Are you running with the new built-in toolchains or a custom Python toolchain?

    Rolling back setuptools may be a short-term solution, but that only hides the issue and we should be able to upgrade it freely in the future (particularly as new Python releases come out that could start to have issues with a very old setuptools version). In this particular case for this issue: there's nothing that rules_python can fix (besides maybe updating the demo code I linked), as it's my local custom toolchain layout that was breaking distutils. It just so happens that a fair number of us have adopted and adapted that original demo code, so are seeing similar bugs.

  12. groodt commented on Jun 12, 2022

    @groodt
    Contributor

    My issue is now resolved after upgrading rules_python to the latest and upgrading the failing dependency to a wheel, rather than relying on the sdist. Relying on the sdist was fragile to setuptools and distutils changes as you mention.

    I no longer believe I was seeing the same issue, so I think we still need to see a repro from the original poster to be certain (although I do think @aaliddell has the best explanation so far).

  13. hrfuller commented on Jun 13, 2022

    @hrfuller
    Contributor

    Agreed the @aaliddell bisect makes sense to me.

  14. groodt commented on Jul 3, 2022

    @groodt
    Contributor

    @davians12 Can you confirm whether @aaliddell fix solves your issue?

  15. added
    Can Close?Will close in 30 days if there is no new activity
    on Jul 3, 2022
  16. aaliddell commented on Jul 3, 2022

    @aaliddell
    Contributor

    A useful closure here could be to update the example python build toolchain here to fix the symlink being inside the source tree.

  17. removed
    Can Close?Will close in 30 days if there is no new activity
    on Jul 3, 2022
  18. garymm commented on Aug 29, 2022

    @garymm
    Contributor

    For me this was fixed on MacOS by version 0.12.0. From the release notes, I'm guessing fixed by #809

  19. added
    Can Close?Will close in 30 days if there is no new activity
    on Sep 10, 2022
  20. github-actions commented on Oct 10, 2022

    @github-actions
    Contributor

    This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Can Close?Will close in 30 days if there is no new activity

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions