Skip to content

rules/python: Option to generate .pyc files for py_library. #1761

Closed

Description

@adam-azarchs

Description of the feature request:

Add options on one or more of the py_library rule itself, the python toolchain configuration, or the bazel command line, to generate python bytecode files as part of the build.

In addition to the basic on/off option, it might be useful to include options to

  1. Control the "optimization" level (for what that's worth)
  2. Leave the source .py file out of the build outputs - the .pyc-only use case.

What underlying problem are you trying to solve with this feature?

When python loads a module, it parses the source code to an AST, which it then attempts to cache in bytecode format. Building those as part of the build process solves a few issues:

  1. To quote the documentation for the py_compile module:

    Though not often needed, this function can be useful when installing modules for shared use, especially if some of the users may not have permission to write the byte-code cache files in the directory containing the source code.

    Particularly in the case where bazel is being used to build a tarball of code that includes (but may not be limited to) python, and might then be deployed somewhere that's read-only to most users, it would be useful to be able to include these precompiled bytecode files.

  2. The attempt to compile the bytecode files would fail on syntactically-invalid python code, which is probably a good thing for catching failures earlier on in the build process.

  3. Having .pyc files available improves application startup time. Especially for large python codebases, if some module is transitively imported from thousands of unit tests, currently each of those tests would end up re-parsing the python source file, which is a waste of time. Having the .pyc files is also helpful for improving startup times for "serverless" platforms.

  4. The .pyc files can be substantially smaller than the source files. For situations where application distribution size is important, e.g. "serverless" platforms, this can matter.

  5. Some people place value on the marginal degree of obfuscation and tamper resistance offered by .pyc-only distributions. While reverse-engineering the source from a .pyc file isn't hard, it's also not nothing.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 5.3.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

References/notes:
https://docs.python.org/3/library/py_compile.html
The default compilation mode is TIMESTAMP; this is probably a bad idea. In a bazel build we'd probably want to use UNCHECKED_HASH. Would also need to ensure that the embedded path name was appropriately relative.

From PEP-3147:

Python will still support pyc-only distributions, however it will only do so when the pyc file lives in the directory where the py file would have been, i.e. not in the pycache directory. pyc file outside of pycache will only be imported if the py source file is missing.

This means that in the case where the .py file is still being included, the output path would need to depend on the python interpreter version. This probably would require an attribute to be added to py_runtime for that purpose.

Activity

  1. comius commented on Oct 6, 2022

    @comius
    Contributor
  2. rickeylev commented on Oct 6, 2022

    @rickeylev
    Collaborator

    This is probably fairly easy to do by adding a precompiler attribute to the toolchain that the rules can use.

    The tricky part I'm not sure of is what that attribute points to. It can't point to a py_binary itself, since that would be a circular dependency. But we have to use Python to run py_compile somehow.

    Maybe allow it to be an executable or a single py file? If it's an executable, run it. If it's a py file, then run "$interpreter $file". This allows creating a pre-built executable if desired, while also allowing it to Just Work.

    (this again touches on the problem of how there's several cases where py-binaries want to be used as part of the toolchain)

    In a bazel build we'd probably want to use UNCHECKED_HASH

    Probably, yes. CHECKED_HASH is somewhat appealing, too, though, because it allows you to modify the source file and see the new behavior without having to rebuild, though.

    I'm +1 on defaulting to one and having an option for the other. Maybe base the default on -c opt so optimized builds used unchecked_hash by default, while fastbuilds use CHECKED_HASH by default. I can easily see a server not caring about this difference, while a CLI app would.

    I also like the idea of a library-level control here somehow, too. A scenario we've enountered with large packages (e.g. tensorflow), is the sheer number of modules has its own overhead; pyc helps, but being as how modifying them is the exception, not the rule, making them check their hash is wasted work. We'll have to figure out how a target-level and global-flag-level option should interact, though (which has precedence?).

    This means that in the case where the .py file is still being included, the output path would need to depend on the python interpreter version. This probably would require an attribute to be added to py_runtime for that purpose.

    Yeah, the details of the pyc infix aren't well understood to me. I think PEP 488 is another half of the puzzle. I swear I thought there was a part of the specification that included the "compile mode" of Python itself, too (e.g. debug for non-debug), but I don't see that mentioned.

    We also have to make sure that the generated name is recognized by the runtime that will be used. I think this is controlled by BYTECODE_SUFFIXES? One option is to simply stick this onto the toolchain, too. Another would be to pass the necessary suffix to the stub template so it can configure the runtime at startup.

    Additionally, the output path depends on if the original .py files is included or not.

    The Major.Minor Python version is in the --python_version flag.

  3. adam-azarchs commented on Oct 7, 2022

    @adam-azarchs
    Author

    The tricky part I'm not sure of is what that attribute points to. It can't point to a py_binary itself, since that would be a circular dependency. But we have to use Python to run py_compile somehow.

    Maybe allow it to be an executable or a single py file? If it's an executable, run it. If it's a py file, then run "$interpreter $file". This allows creating a pre-built executable if desired, while also allowing it to Just Work.

    (this again touches on the problem of how there's several cases where py-binaries want to be used as part of the toolchain)

    Yeah, this is kind of a similar problem as the one faced for the coverage tool in bazelbuild/bazel#15590. However, there's an important difference: py_compile is part of the python standard library. Given that, I don't really see a need to offer an option to customization of the compilation tool - bazel could just hard-code running $interpreter -m py_compile, which as far as I can tell would work for all of CPython, PyPy, and MicroPython at least (though I've only tried with CPython).

    Probably, yes. CHECKED_HASH is somewhat appealing, too, though, because it allows you to modify the source file and see the new behavior without having to rebuild, though.

    I'm +1 on defaulting to one and having an option for the other. Maybe base the default on -c opt so optimized builds used unchecked_hash by default, while fastbuilds use CHECKED_HASH by default. I can easily see a server not caring about this difference, while a CLI app would.

    Absolutely! However, that's something you can handle with select, so it doesn't really have to impact the design much.

    I think the options we'd want exposed look something like

    1. No compilation.
    2. CHECKED_HASH
    3. UNCHECKED_HASH
    4. pyc-only

    As well as the mostly orthogonal choices for "optimization" level. (which I will continue to put in quotes. It's really too bad that there isn't an optimization level which strips out docstrings, which you don't need in a server context, but leaves in assertions, which might be a bit dangerous to strip out)

    I also like the idea of a library-level control here somehow, too. A scenario we've enountered with large packages (e.g. tensorflow), is the sheer number of modules has its own overhead; pyc helps, but being as how modifying them is the exception, not the rule, making them check their hash is wasted work. We'll have to figure out how a target-level and global-flag-level option should interact, though (which has precedence?).

    This is why I suggested pre-compilation should be an attribute on the py_library rule, as well as possibly the ability to set a default at the toolchain level. For one thing, you might want to have some things be distributed as pyc-only, while others might want to include source. You might also want to be able to control the "optimization" level on a per-target basis. I think the safe approach there would be to just say local specification takes precedence over global specification - I can't really imagine a situation where someone would set an override on a library and we'd not want to trust that they had a good reason to do so. At a nitty-gritty API level, however, it does mean we have to be careful about the value of the attribute one sets for this; one might think that compile_mode = None would be a default of "use the toolchain default," but that would catch someone by surprise if they (somewhat reasonably) read that as meaning "no compilation". Easy enough to handle - just make the default for the attribute be e.g. "from_toolchain" or whatever.

  4. rickeylev commented on Oct 10, 2022

    @rickeylev
    Collaborator

    Yeah, this is kind of a similar problem as the one faced for the coverage tool in bazelbuild/bazel#15590. However, there's an important difference: py_compile is part of the python standard library. Given that, I don't really see a need to offer an option to customization of the compilation tool - bazel could just hard-code running $interpreter -m py_compile, which as far as I can tell would work for all of CPython, PyPy, and MicroPython at least (though I've only tried with CPython).

    I think invoking the interpreter directly like that is a workable intermediate solution. So I'm +1 on doing that for now. I'm fairly certain we need a custom program to ensure deterministic behavior and pass the necessary info, though. The cli of -m py_compile isn't sufficient.

    However, I need to make a few architectural points about why such solutions make it hard or impossible to take full advantage of Bazel's processing model and features.

    As some concrete examples:

    When using embedded Python, there's no need for a standalone interpreter, so building one simply for precompilation is unnecessary work. (This is particularly salient to me because that's my primary way of building Python programs).

    Precompilation is a near perfect fit for using persistent workers, which would eliminate runtime startup overhead. This requires special support by the executable invoked.

    Precompilation must run in the exec configuration. This means the runtime itself may be built differently than the runtime actually used, may be run remotely, and/or run on an entirely different platform. So more steps have to be taken to ensure such details don't leak into the output. For example, the launcher script does several things to prevent system settings from affecting the runtime which have to be re-implemented.

    Similarly, the exec configuration usually emphasizes time-to-build over runtime-efficiency. This makes sense for typical user build tools, but less so for something like precompilation; precompilation is well defined and it's rate of invocation scales with the size of the overall build, ie the larger your transitive closure, the more it runs, so you're probably going to benefit from the precompiler (and runtime) being built with optimizations enabled. Such optimizations can extend beyond simply e.g. pyc -O levels, and into C-level optimizations that affect the runtime itself; these optimizations are hard to apply during a build, but are easy to apply to to an prebuilt executable which is later run.

    This isn't to say simply doing "$runtime -m py_compile" is bad or wrong, just that it leaves a lot sitting at the Bazel Buffet Table.

    As well as the mostly orthogonal choices for "optimization" level. (which I will continue to put in quotes. It's really too bad that there isn't an optimization level which strips out docstrings, which you don't need in a server context, but leaves in assertions, which might be a bit dangerous to strip out)

    Srsly, right? This has bugged me for years! It's the sort of thing a custom precompiler executable could do, though. This also makes me think, if we exposed an optimization level, it'd probably be better done as a list of values instead of a simple number.

    I think the safe approach there would be to just say local specification takes precedence over global specification - I can't really imagine a situation where someone would set an override on a library and we'd not want to trust that they had a good reason to do so.

    I agree. A case we have internally is that protocol buffers (which are handled by a special rule) are compiled by default. This is pretty akin to a library-level setting.

    At a nitty-gritty API level, however, it does mean we have to be careful about the value of the attribute one sets for this; one might think that compile_mode = None would be a default of "use the toolchain default," but that would catch someone by surprise if they (somewhat reasonably) read that as meaning "no compilation". Easy enough to handle - just make the default for the attribute be e.g. "from_toolchain" or whatever.

    I agree.

  5. adam-azarchs commented on Oct 10, 2022

    @adam-azarchs
    Author

    Worth noting mostly for the sake of code archeology that there's this comment in the code currently: https://github.com/bazelbuild/bazel/blob/6d72ca979f8cf582f53452d5f905346e7effb113/src/main/java/com/google/devtools/build/lib/rules/python/PythonSemantics.java#L69-L71
    however as far as I can tell the implementation https://github.com/bazelbuild/bazel/blob/9d0163002ca63f4cdbaff9420380d72e2a6e38b3/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java#L117-L120
    simply copy the sources unaltered. But for what it's worth I'm guessing that at some point in the past someone had an implementation, or at least and attempt at one, probably Google-internal-only, for at least some of this functionality.

  6. antonio-arbo commented on Feb 27, 2023

    @antonio-arbo

    Thanks for filing & tracking!
    It would be great to have this option, particularly since this seems to render x-machine remote cache in bazel quite less effective since each machine has its own timestamp of the pyc. So it would be much better to have a more stable and safer hash instead of just timestamps.

    Do you have a suggestion of a workaround for the time-being?

    BTW, for others looking into this. A simple bazel clean -expunge and then rebuild will show many remote cache misses. Confirm it looking into the exec logs as in https://bazel.build/remote/cache-remote .

  7. Ryang20718 commented on Feb 15, 2024

    @Ryang20718

    I would very much <3 to see this feature.

    we have a lot of heavy imports :(

    Some of our heavy hitter imports bog down both test & runtime (mainly test)

    One of the pain points right now is that we use pytorch in our bazel monorepo. However, because we import it in so many of our libraries, it incurs a large import time overhead (3.8 seconds).

    With pycache, it drops down to ~2.5 seconds.

    Is the general idea to generate pyc files once when all external third party repos are fetched at analysis time and then to ingest these files as input in py_library?

    Would love some pointers on where to start (even if it's just a hacky prototype 😅 )

  8. rickeylev commented on Feb 15, 2024

    @rickeylev
    Collaborator

    I'm going to transfer this issue to the rules_python repo because, as of rules_python 0.31.0, we've enabled the rules_python-based implementation of the rules by default for Bazel 7+. This basically means this feature won't land in the Bazel builtin rules, but would instead be in the rules_python implementation.

  9. transferred this issue frombazelbuild/bazelon Feb 15, 2024
  10. rickeylev commented on Feb 15, 2024

    @rickeylev
    Collaborator

    Would love some pointers on where to start (even if it's just a hacky prototype 😅 )

    Can do! I've had parts of this mentally sketched out, but haven't thought it through entirely. This is already partially wired into the rules_python implementation, too.

    There's a stub function, maybe_precompile() src that gets called. As output, it returns the new set of sources a library should use, which is either (a) the py files (no precompiling), or (b) both the py files and pyc files, or (c) just the pyc files.

    All this function really has to do is loop over the sources, run a tool to generate the pyc files, and return them. The important thing is that it generates deterministic output (i.e. no timestamp based pyc files). It'll probably need some flags to control behavior a bit, as discussed, but lets focus on a prototype first. Anyways, the core code of this function is pretty simple:

    pycs = []
    for src in src_files:
        # Generating a file in another package is an error, so we have to skip
        # such cases.
        if ctx.label.package != src.owner.package:
            continue
        pyc = ctx.actions.declare_file(..., sibling = src)
        ctx.actions.run(
            executable = ...,
            inputs = [src],
            arguments = ... + [src.path, pyc.path],
            outputs = [pyc],
            mnemonic = "PyCompile",
            progress_message = "Compiling Python %{input}",
            toolchain = ...,
        )
        pycs.append(pyc)
    return pycs
    

    That's the first part, which is easy.

    The second part is defining the tool that is run to generate the pyc. For the purposes of experimenting, you can add an implicit attribute on py_library pointing to some executable to run. The tricky part here is the executable it points to can't be a regular py_binary -- that would result in a circular dependency. For experimenting, define it how you like to work out details (e.g. a sh_binary that just calls the system python would suffice for testing purposes).

    Within Google, we prebuild a binary and use an implicit attribute. A prebuilt binary is actually a good thing in this case, but not required. It allows neat things like building an binary with optimizations, having a generic binary that can generate byte code for any python version, building a cc_binary that embeds python, or heck, you could implement a byte code compiler in Rust or whatever if you really wanted to. I'm getting ahead of myself. Anyways.

    It's probably best to just start with invoking something like python precompile.py <src> <pycout>. That's going to be pretty close to how a non-bazel python works.

    Using an implicit attribute, however -- that won't work well in the Bazel world where the environment isn't as tightly controlled and where there are many more platform configurations to worry about.

    In the open Bazel world, this precompiler tool needs to come from a toolchain. This is because we need toolchain resolution to find a tool that matches the target config (the target python version we need to generate byte code for) that can run on one of our available execution platforms. A basic implementation that re-used the same runtime would look something like this, I think:

    toolchain(
      name = "precompiler_3_10_linux_toolchain",
      target_compatible_with = [
        "@rules_python//python/config_settings:is_python_3.10",
      ],
      exec_compatible_with = ["@platforms//os:linux"],
      toolchain = ":precompiler_toolchain_impl",
      toolchain_type = "//python:precompiler_toolchain_type",
    )
    
    py_precompiler_toolchain(
      name = "precompiler_toolchain_impl",
      interpreter = ":bin/python3.10",
      precompiler_src = "@rules_python//tools/precompiler:precompiler.py"
    )
    
    py_precompiler_toolchain = rule(..., attrs = {
      "interpreter": attr.label(cfg="exec"),
      "precompiler_src": attr.label(cfg="exec"),
    })
    
    # In the maybe_precompile function
    ctx.actions.run(
      executable = ctx.toolchains["//python:precompiler_toolchain_type"].interpreter
      args = [ctx.toolchains["//python:precompiler_toolchain_type"].precompiler_src] + [src.path, pyc.path],
      toolchain = "//python:precompiler_toolchain_type",
      ...
    )
    

    (there's variations of the above that would also work, but thats the gist).

  11. Ryang20718 commented on Mar 2, 2024

    @Ryang20718

    Was prototyping a bit. I really appreciate all the pointers here!

    I used the system python interpreter to prototype for now and noticed some weird behavior and perhaps I'm missing something fundamental here, but when bazel "compiles" these python files into pyc, sometimes, we run into the case where the sandbox only contains the directory bazel-out. cding into the sandbox debug path, we see only bazel-out exists and no "external" directory exists for external/pip_grpcio/site-packages/grpc/framework/interfaces/face/utilities.py

      (cd /dev/shm/bazel-sandbox.2566462e83a3701bdcf2405eb4d15ec8f4c1eed3744dd6958e7c02b8ef25ffcf/linux-sandbox/153/execroot/test_pyc && \
      exec env - \
        TMPDIR=/tmp \
      /home/ryang/.cache/bazel/_bazel_ryang/install/14fb027596f626f2526df4873ea20b8b/linux-sandbox -t 15 -w /dev/shm -w /dev/shm/bazel-sandbox.2566462e83a3701bdcf2405eb4d15ec8f4c1eed3744dd6958e7c02b8ef25ffcf/linux-sandbox/153/execroot/test_pyc -w /tmp -e /tmp -S /dev/shm/bazel-sandbox.2566462e83a3701bdcf2405eb4d15ec8f4c1eed3744dd6958e7c02b8ef25ffcf/linux-sandbox/153/stats.out -D /dev/shm/bazel-sandbox.2566462e83a3701bdcf2405eb4d15ec8f4c1eed3744dd6958e7c02b8ef25ffcf/linux-sandbox/153/debug.out -- /bin/bash -c '/bin/python3.10 -m py_compile external/pip_grpcio/site-packages/grpc/framework/interfaces/face/utilities.py')
     ....
     
    
     ls -a
    .  ..  bazel-out
     
    

    Debugging this prototype, we see src.path for this situation is external/pip_grpcio/site-packages/grpc/framework/interfaces/face/utilities.py and pyc.path is bazel-out/k8-opt/bin/external/pip_grpcio/site-packages/grpc/framework/interfaces/face/utilities.cpython-310.pyc

    Is there something fundamental that I'm missing?

    Prototype:

    def maybe_precompile(ctx, srcs):
        pycs = []
        for src in srcs:
            if "site-packages" not in src.path:
                pycs.append(src)
                continue
    
    
            basename = src.basename
            dirname = src.path.rsplit("/", 1)[0].split("/", 2)[2]
    
            pyc_out = dirname + "/__pycache__/" + basename.replace(".py", ".cpython-310.pyc")
            pyc = ctx.actions.declare_file(pyc_out)
            ctx.actions.run_shell(
                outputs = [pyc],
                command = "/bin/python3.10 -m py_compile " + src.path
            )
    
            pycs.append(pyc)
        return pycs
        ```
    
  12. 20 remaining items

  13. adam-azarchs commented on Mar 21, 2024

    @adam-azarchs
    Author

    A trickier case to handle where you'd want to use UNCHECKED_HASH is unit tests, for which bazel should always be guaranteeing the .pyc is up to date (assuming they're running in either sandbox or remote), so you might as well take that optimization.

  14. Ryang20718 commented on Mar 25, 2024

    @Ryang20718

    Was doing some prototyping to pipeline this pyc_mode through to PyInfo and had a couple of questions

    I notice that in areas wheremaybe_precompile is used, PyInfo is generated with the outputs of maybe_precompile. I.e here

    Assuming we'd want the end user to be able to do something like

    py_library(
        name = "blah",
        srcs = ["a.py"],
        ...
        pyc_mode = "pyc_only",
    )
    

    Would we need to propagate from PY_SRCS_ATTRS? Or is there a way to do this solely via PyInfo?

    1. For creating a pre-compiler toolchain, after reading some docs + Richard's suggestions, I was wondering if there's a way to pass in the interpreter from the current python toolchain when instantiating this precompiler toolchain?

    Currently, I just use a sh_binary which wraps a local python, but ideally it would match the python toolchain defined?

    def _py_precompiler_toolchain_impl(ctx):
        return [platform_common.ToolchainInfo(
            interpreter = ctx.attr.interpreter,
            precompiler_src = ctx.attr.precompiler_src,
            python_version = ctx.attr.python_version,
        )]
    
    py_precompiler_toolchain = rule(
        doc = """
        This rule exists so that the precompiler toolchain can be set for precompiling python files to pyc without.
        This cannot be a regular py_binary, otherwise there will be a circular dependency.
        """,
        implementation = _py_precompiler_toolchain_impl,
        attrs = {
            "interpreter": attr.label(cfg="exec"),
            "precompiler_src": attr.label(cfg="exec", default = Label("//tools/precompiler:precompiler_py")),
            "python_version": attr.string(doc = "python version. I.e 310 for python 3.10. Used for generating magic tag for pyc file", mandatory = True),
        },
    )
    
    
    1. Multiple Python Versions. we currently haven't yet migrated to bzlmod. we also aren't using multiple python versions so this comment may be out of date.

    Does bzlmod support pytest with multiple pip/python versions? I see there's this comment here

  15. rickeylev commented on Apr 16, 2024

    @rickeylev
    Collaborator

    Coincidentally, we have some internal work in this area planned, so I thought I'd post some of our analysis that seemed relevant.

    Note that this isn't documenting a decision. It's merely trying to inform from looking at the use case we're trying to solve (in brief, precompiling enabled by default) and the constraints we have.


    A precompile attribute with 5 values seems very useful:

    • all: include both py and pyc files
    • only: include only pyc files
    • none: include only py files
    • inherit: pick one of the above from a flag
    • always: always include a pyc file, but decide if py is included from a flag

    It seems like there are 3 flags to decide behavior?

    • --precompile_default: used when a target doesn't have a value set (or when its inherit). It would have values:

      • all, only, none: same as for the attribute
      • generated_only: For generated files, acts as "only". For source files, acts as "none".
      • generated_all: For generated files, as as "all", for source files, as as "none"
    • --precompile_force: Like --precompile_default but overrides target level settings. Probably the same value as --precompile_default. (The use cases for this seem valid but niche, see below).

    • --precompile_force_retain_sources: Forces that py sources are retained. e.g. precompile=only acts like precompile=all (mostly an alternative to precompile_force, see below)


    re: flag vs attribute precedence: this is a really tough question. We could come up with reasonable scenarios where both are sensible behaviors. Overall, the attribute taking precedence seems like the better default behavior. The main case for having the flag take precedence is: what if a target you don't own sets precompile=only (having just pyc makes sense if it just generates code), but you have to deploy to an environment where that pyc won't be valid (i.e. your build is for Python 3.10, but you deploy to something with python 3.9). This is somewhat of an edge case for us, but seems like it would be much more common in OSS. This case is also where the --precompile_force_retain_sources flag comes from: by forcing sources to be kept, the invalid pyc wouldn't be an issue in practice.


    The above case also implies that there's a distinction between binaries and libraries in this regard (i.e. a binary level-setting applies transitively). As a general rule, if there's a global flag to set the default or force the state, then it probably should also be a binary-level attribute. It's not always possible or desirable to set a global flag, especially if there's just a handful of targets that need it.

    That said, a binary level setting also comes with problems. As an example, given:

    • Binary setting has precedence
    • binary sets precompile=all
    • library elsewhere sets precompile=only
    • Transitions are not allowed to be used (performance reasons)

    Then the problem is the library can't directly see what the binary needs, and only outputs foo.pyc, when it should have output both foo.py and __pycache__/foo.pyc. We think an aspect can address this, but its illustrative of some of the caveats and complex interactions that can happen (and also how transitions make it easy for an unfortunately high cost).

  16. rickeylev commented on Apr 16, 2024

    @rickeylev
    Collaborator

    Would we need to propagate from PY_SRCS_ATTRS? Or is there a way to do this solely via PyInfo?

    I'm not sure I understand the question. What goes into PyInfo is derived from the target attributes (srcs and deps, usually). The core logic, when unrolled, should basically be:

    PyInfo(
      direct_sources = ctx.files.srcs,
      transitive_sources = direct_sources + [d[PyInfo].transitive_sources for d in deps],
      direct_pyc = get_precompile_outputs(direct_sources),
      transitive_pyc = direct_pycs + [d[PyInfo].transitive_pyc for d in deps]
      ...
    )
    

    Does that answer your question?

    a way to pass the current python toolchain interpreter in

    I think you're hinting at the issue of you need an exec-config interpreter, but the python toolchain only has a target-config interpreter?

    I think your 3 basic options are:

    1. Use manually defined exec groups, see https://bazel.build/extending/exec-groups. A helper target is probably necessary, e.g. //python:current_py_toolchain or similar.
    2. Use auto exec groups, see https://bazel.build/extending/auto-exec-groups. I think this requires a separate toolchain type with cfg=exec parts.
    3. Create a "py_exec_interpreter_binary" rule and have an implicit, cfg=exec, dependency on it. All it does is create an executable that acts like the interpreter. This is akin to what you're doing with sh_binary, except a dedicated rule to do it instead.

    I think a combination of (2) and (3) is probably the best? (2) invokes toolchain resolution, which permits the target config to influence selection (i.e. allows the target config pythonversion=3.10 to select a toolchain that can generate 3.10 bytecode). (3) allows abstracting how the precompiler binary is implemented (i.e. it could be a pre-compiled binary or simply lookup the "regular" interpreter).

    ideally it'd match the current toolchain

    Yes, but recall how toolchain resolution and matching works: it's taking the key:value constraints and looking over lists to find something that matches. I say this because it sounds like you might be thinking/desiring this: regular_toolchain = match(X, "regular"); precompile_toolchain = create_precompile_toolchain_from(regular_toolchain), but that isn't how toolchains operate. It's more like this: regular_toolchain = match(X, "regular"); precompile_toolchain = match(X, "precompile").

    Does bzlmod support pytest with multiple pip/python versions?

    Yes, it supports multiple versions. That comment is out of date.

  17. adam-azarchs commented on Apr 18, 2024

    @adam-azarchs
    Author

    Then the problem is the library can't directly see what the binary needs, and only outputs foo.pyc, when it should have output both foo.py and __pycache__/foo.pyc. We think an aspect can address this, but its illustrative of some of the caveats and complex interactions that can happen (and also how transitions make it easy for an unfortunately high cost).

    This can be avoided if we're willing to complicate the PyInfo to separately track sets of files for each use case as I suggested in #1761 (comment). This is probably not a great approach, however, as there are probably a lot of existing rules out there that propagate PyInfo and might start silently dropping these additional fields.

  18. rickeylev commented on May 12, 2024

    @rickeylev
    Collaborator

    I've been working on adding this functionality. You can see the WIP here: main...rickeylev:rules_python:precompile.toolchain

    Everything appears to work, so I've started cleaning it up.

    The basic gist of what I have implemented is:

    • A new toolchain to carry an exec-config interpreter and precompiler tool.
    • The precompiler can be either run with the exec-config interpreter, or be a prebuilt binary.
    • The precompiler that comes with rules_python is a fully multiplexed, sandbox-aware,
      persistent worker. (I started on this path for fun, but then it turned out to be a
      huge performance benefit -- 60+ seconds vs 4 seconds when building the 2000 file
      bzlmod example)
    • Provider fields PyInfo.transitive_pyc_files, PyInfo.direct_pyc_files
    • Several new attribute and flags to control things:
      • precompile=inherit|enabled|disabled|if_generated_source: controls if a target
        performs precompilation.
      • --precompile=auto|enabled|disabled|if_generated_source|force_enabled|force_disabled
      • precompile_invalidation_mode=auto|checked_hash|unchecked_hash: determines what invalidation mode to use.
      • precompile_optimize_level=<int>: Controls the optimization level.
      • --precompile_source_retention=auto|keep_source|omit_source|omit_if_generated_source: controls if the py is included in the output or not.
      • (binaries/tests only) pyc_collection: controls if binaries explicitly include pyc files
        from dependencies. This derives from the Google situation, where controlling behavior
        at the binary level makes it easier to incrementally roll out the change.
      • --precompile_add_to_runfiles=auto|always|decided_elsewhere: controls if the pyc files
        are added to the runfiles of the generating target (e.g. if py_library adds its pyc
        files to the runfiles it returns). This is used in combination with pyc_collection.

    The two basic ways to use precompilation are:

    1. Each target (e.g. py_library) compiles its files and they're included in runfiles and it Just Works. This will eventually be the default mode.
    2. Libraries can create compiled files, but not include them in the runfiles. Binaries
      can then opt-into including the pyc files. This is more for the Google situation, but
      really any situation where enabling precompiling at large doesn't Just Work.

    If you like optimizing Python code, then optimizing the precompiler tool might be fun. Unfortunately, precompiling adds a decent amount of overhead. As an example, the bzlmod example has about 2,000 py files. This takes about 60+ seconds to precompile as a regular action (1 py file per action) 🙁.

    So I implemented a second precompiler that is a multiplexed, sandbox-aware, persistent worker. This reduced build time from 60s to about 4 seconds (yes, single-digit). I think we can do better, but I haven't explored where the bottleneck is. If we can change it to be a prebuilt binary, then we'd have more options for optimizing it, too.

  19. adam-azarchs commented on May 14, 2024

    @adam-azarchs
    Author

    A few observations:

    Persistent worker

    This approach worries me a bit because there's a lot of rough edges around using a persistent worker with remote execution (which my team uses exclusively).

    Python interpreter startup time is a lot; it's not surprising that you get significant benefits from a persistent worker, especially since AFAKT the implementation for py_compile is to actually fully import the module, which ends up processing all of its transitive imports etc. Could probably improve on that very significantly with a purpose-built binary. I haven't actually looked into how hard that would be to write, but at first glance at least if you're allowed to link libpython it looks like you'd need to

    1. PyImport_GetMagicNumber for the header
    2. Add the flags
    3. Add source hash. This part is maybe the tricky part, seems like _imp_source_hash is the right call to make here but that's a private symbol. You could go through the python extension module _imp.source_hash but at that point I'm not sure how much benefit from avoiding a full interpreter is preserved...
    4. call Py_CompileStringObject to get the code object
    5. PyMarshal_WriteObjectToString to serialize it

    pyc-only mode

    I'm not sure this works in if precompile="disabled" is set on any dependencies. I think you can get around that by adding the sources into the "pyc" depset in that situation, though that muddies the meaning a bit.

    You're also not supporting the use case of permitting specific libraries to be pyc-only without setting it overall. This could be done; similarly to precompile=disabled you'd need to add the pyc to the normal sources as well as as pycs in order to support cases that were not collecting pyc.

  20. rickeylev commented on May 14, 2024

    @rickeylev
    Collaborator

    re: persistent worker: Yes, all good points. The precompiler works as a non-worker, regular worker, or multiplexed worker, and there's a flag to set the execution_requirements for the precompile action.

    re: py_compile fully imports: It doesn't import (execute) the py code. It just uses the importlib libraries to read the file and compile it to pyc. I found that surprising, but maybe not so much if this is closely coupled to the import system's processing of things (e.g. _imp_source_hash being part of import.c).

    re: linking with libpython: Yes, that'd be awesome. I'd love to get a cc_binary that statically links with libpython working. For something like the precompiler, which is a single file that only uses the stdlib, such a thing seems doable.

    re: library-level pyc-only: Ah yeah, I forgot about that. That should be doable with target-level attribute to control that.

  21. adam-azarchs commented on May 14, 2024

    @adam-azarchs
    Author

    I forgot that you can't actually choose to be pyc-only at the py_binary level because the file naming requirements are different for pyc-only vs. py+pyc. So that decision needs to be done at the py_library level, but can have a global default at least.

    I'll maybe take a time-boxed crack at trying to get such a cc_binary working tomorrow.

  22. rickeylev commented on May 15, 2024

    @rickeylev
    Collaborator

    re: remote builds, performance, persistent workers: A thought: Right now, each source file is processed with a separate action. Some performance could be regained with non-worker execution by sending all of a target's sources to a single invocation. This would at least change the overhead to once-per-target instead of once-per-file. For targets with many sources (e.g. something from pypi), this would probably help a decent amount.

  23. adam-azarchs commented on May 15, 2024

    @adam-azarchs
    Author

    This would at least change the overhead to once-per-target instead of once-per-file.

    One of our repos using remote execution:

    $ bazel query 'kind(py_library, //...) + kind(py_test, //...) + kind(py_binary, //...)' | wc -l
    2660
    
  24. adam-azarchs commented on May 15, 2024

    @adam-azarchs
    Author

    Didn't really have time to figure out a truly portable way to do the source hashing, but I did try a different, much simpler python implementation:

    from sys import argv
    import py_compile
    
    src_path, src_name, dest, optimize, checked = argv[1:]
    
    py_compile.compile(
        src_path,
        cfile=dest,
        dfile=src_name,
        optimize=int(optimize),
        invalidation_mode=(
            py_compile.PycInvalidationMode.CHECKED_HASH
            if bool(int(checked))
            else py_compile.PycInvalidationMode.UNCHECKED_HASH
        ),
    )

    This avoids argparse importing of all of the stuff you need for a persistent worker, and seems to reliably take less than half the time as your precompile.py when invoked on a single file. This isn't that surprising; what argparse is doing is pretty expensive if you're going to be doing it a huge number of times, though it's worth it for a longer-lived process.

    $ time (for f in $(git ls-files '*.py'); do python3 tools/precompiler/precompiler.py --src $f --src_name $f --pyc /tmp/temp2.pyc --invalidation_mode UNCHECKED_HASH --optimize=0; done)
    
    real    0m10.133s
    user    0m8.744s
    sys     0m1.381s
    $ time (for f in $(git ls-files '*.py'); do python3 tools/precompiler/precompiler2.py $f $f /tmp/temp.pyc 0 0; done)
    
    real    0m3.492s
    user    0m2.867s
    sys     0m0.600s

    This does suggest that the persistent worker implementation should at the very least be kept separate from the single-file implementation. Or at least should process imports lazily as-needed.

  25. rickeylev commented on May 16, 2024

    @rickeylev
    Collaborator

    Oh, nice analysis. I see similar. All those unused imports add up.

    re: lazy imports: Yeah, I'll do this, at the least. This is an easy change to make.

    re: splitting worker impl into another file: That's a good idea, too; it would avoid having to parse all the worker-related code. It'll have to wait for another PR, though.

    The argparse import itself isn't too bad, so I'm going to keep the --flag based calling convention. Actually, its more an @file convention with the args within in. We don't know if the action will use a worker or not, so the args have to always be passed to the param file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions