Skip to content

Add mirrors for pip_install_dependencies #798

Closed

Description

@kgreenek

🚀 feature request

Relevant Rules

pip_install_dependencies macro from @rules_python//python/pip_install/repositories.bzl

Description

In our CI, we frequently see download errors caused by failing to download dependencies from rules_python. For example, the following bazel error:

Error in download_and_extract: java.io.IOException: Error downloading [https://files.pythonhosted.org/packages/76/0a/b6c5f311e32aeb3b406e03c079ade51e905ea630fc19d1262a46249c1c86/click-8.0.1-py3-none-any.whl] to /tmp/bazel/c75bfb83ea925bcd720c63452511dc13/external/pypi__click/temp13170534750999975287/click-8.0.1-py3-none-any.whl.zip: connect timed out

Our solution is to fork the @rules_python//python/pip_install/repositories.bzl file in our repo, and add additional mirror URL's in case files.pythonhosted.org is failing. This works great, but it is a bit painful to maintain, since we have to update it to keep it in sync whenever updating rules_python.

Describe the solution you'd like

Add additional mirrors for external dependencies to make the downloads more reliable.

Another idea would be to add the ability to specify our own mirrors to the call to pip_parse without forking the repositories.bzl file.

Describe alternatives you've considered

See above.

Activity

  1. driftregion commented on Sep 5, 2022

    @driftregion

    @kgreenek , is it possible to use --distdir as a workaround in your situation? I'm using this to eliminate network requests for dependencies in my projects including those listed in @rules_python//python/pip_install/repositories.bzl

    https://docs.bazel.build/versions/main/external.html#offline-builds

    http_archive (used by pip_install_dependencies) can be instructed to search locally before making a network call. (details: bazelbuild/bazel#9265 (comment))

    I realize this isn't exactly what you're asking for, but it might offer some improvement over the current situation.

  2. kgreenek commented on Sep 6, 2022

    @kgreenek
    Author

    That looks like a helpful suggestion! I will look into using this.

  3. github-actions commented on Mar 5, 2023

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
    Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

  4. github-actions commented on Apr 5, 2023

    @github-actions
    Contributor

    This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Can Close?Will close in 30 days if there is no new activity

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions