Skip to content

pip_parse is burdensome for downstream projects #608

Closed

Description

@pauldraper

Typically, to install an upstream project, it looks something like

http_repository(
  name = "example",
  # ...
)

load("@example//:workspace.bzl", "example_repositories")
example_repositories()

However, suppose example uses pip dependencies.

The install must be at minimum:

http_repository(
  name = "example",
  # ...
)

# create pip_parse repo
load("@example://workspace_part_1.bzl", "example_repositories_part_1")
example_repositories_part_1()

# load requirements.bzl from pip_parse repo and call install_deps()
load("@example//:workspace_part_2.bzl", "example_repositories_part_2")
example_repositories_part_2()

This looks weird, and further burdens already lengthy WORKSPACE files.


A solution is to expose generating requirements.bzl. Then the upstream project can commit that to the source tree (probably the same step as generating the requirements lock file today). And then downstream projects can initialize repos with only one step.

This allows the repository creation to happen in one step.

Activity

  1. added a commit that references this issue on Jan 24, 2022
  2. hrfuller commented on Feb 1, 2022

    @hrfuller
    Contributor

    Thanks for the report @pauldraper. I'm open to the solution you proposed of generating and committing a requirements.bzl file. However I'd like to understand if a solution like this macro could work to hide the creation of a dependency on rules_python?

  3. pauldraper commented on Feb 2, 2022

    @pauldraper
    Author

    @hrfuller that definitely could.

    Whatever you used to generate _RULE_DEPS and the rest of that is exactly what I'm looking for.

    (If you did that by hand, then automating that is what I'm looking for.)

  4. hrfuller commented on Mar 13, 2022

    @hrfuller
    Contributor

    _RULE_DEPS was not generated to the best of my knowledge. What I was imagining to tidy up the WORKSPACE file is something like:

    http_archive(
       name = "rules_python",
       ...
    )
    
    load("@rules_python//:macros.bzl", "create_pip_parse_repo")
    create_pip_parse_repo(
       name = "pip_parse_custom",
       extra_pip_args = ["--index-url", ...]
       requirements_lock = "//path/to:requirements_lock.txt"
    )
    

    Would that work? I think its a better solution than committing a generated requirements.bzl file, which would have to be manually regenerated and commited anytime you want to change the python requirements available from requirements.bzl

  5. pauldraper commented on Mar 14, 2022

    @pauldraper
    Author

    You need a second step there.

    You haven't actually created any @pip_boto3 etc repos.

    You'll need another load() and those can't existing in Starlark function bodies.

    I think its a better solution than committing a generated requirements.bzl file, which would have to be manually regenerated and commited anytime you want to change the python requirements available from requirements.bzl

    I understand....but that is how lock files normally work.

    requirements.bzl is a lockfile, with URLs, digests, etc.

  6. alexeagle commented on Mar 14, 2022

    @alexeagle
    Contributor

    Yes and checking in the generated .bzl is how it works in Go as well, https://github.com/bazelbuild/bazel-gazelle#update-repos

  7. alexeagle commented on Mar 14, 2022

    @alexeagle
    Contributor

    Also the two-step execution makes this a lot harder to model as a Module Extension in bzlmod. I had a similar setup for JS and had a lot of trouble which I documented on bazelbuild/bazel#14445
    I'm pretty sure that's resolvable, but I didn't actually get back to that to confirm yet.

    Let's add a mode here where the generator can be run as a tool that writes to a .bzl file in the workspace.

  8. added 5 commits that reference this issue on Mar 15, 2022
    4245614
    89efaaf
    abc3bfb
    6647c6f
    146c8d1
  9. hrfuller commented on Mar 15, 2022

    @hrfuller
    Contributor

    You'll need another load() and those can't existing in Starlark function bodies.
    Ah yeah good point. That changes things.

  10. added a commit that references this issue on Mar 16, 2022
    074b5ef
  11. added a commit that references this issue on Mar 16, 2022
    1b59002
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions