Repository navigation
Description
Description of the feature request:
Add options on one or more of the py_library rule itself, the python toolchain configuration, or the bazel command line, to generate python bytecode files as part of the build.
In addition to the basic on/off option, it might be useful to include options to
- Control the "optimization" level (for what that's worth)
- Leave the source
.pyfile out of the build outputs - the.pyc-only use case.
What underlying problem are you trying to solve with this feature?
When python loads a module, it parses the source code to an AST, which it then attempts to cache in bytecode format. Building those as part of the build process solves a few issues:
-
To quote the documentation for the py_compile module:
Though not often needed, this function can be useful when installing modules for shared use, especially if some of the users may not have permission to write the byte-code cache files in the directory containing the source code.
Particularly in the case where bazel is being used to build a tarball of code that includes (but may not be limited to) python, and might then be deployed somewhere that's read-only to most users, it would be useful to be able to include these precompiled bytecode files.
-
The attempt to compile the bytecode files would fail on syntactically-invalid python code, which is probably a good thing for catching failures earlier on in the build process.
-
Having
.pycfiles available improves application startup time. Especially for large python codebases, if some module is transitively imported from thousands of unit tests, currently each of those tests would end up re-parsing the python source file, which is a waste of time. Having the.pycfiles is also helpful for improving startup times for "serverless" platforms. -
The
.pycfiles can be substantially smaller than the source files. For situations where application distribution size is important, e.g. "serverless" platforms, this can matter. -
Some people place value on the marginal degree of obfuscation and tamper resistance offered by
.pyc-only distributions. While reverse-engineering the source from a.pycfile isn't hard, it's also not nothing.
Which operating system are you running Bazel on?
Linux
What is the output of bazel info release?
release 5.3.0
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
References/notes:
https://docs.python.org/3/library/py_compile.html
The default compilation mode is TIMESTAMP; this is probably a bad idea. In a bazel build we'd probably want to use UNCHECKED_HASH. Would also need to ensure that the embedded path name was appropriately relative.
From PEP-3147:
Python will still support pyc-only distributions, however it will only do so when the pyc file lives in the directory where the py file would have been, i.e. not in the pycache directory. pyc file outside of pycache will only be imported if the py source file is missing.
This means that in the case where the .py file is still being included, the output path would need to depend on the python interpreter version. This probably would require an attribute to be added to py_runtime for that purpose.
Activity
comius commentedon Oct 6, 2022
cc @rickeylev
rickeylev commentedon Oct 6, 2022
This is probably fairly easy to do by adding a
precompilerattribute to the toolchain that the rules can use.The tricky part I'm not sure of is what that attribute points to. It can't point to a py_binary itself, since that would be a circular dependency. But we have to use Python to run py_compile somehow.
Maybe allow it to be an executable or a single py file? If it's an executable, run it. If it's a py file, then run "$interpreter $file". This allows creating a pre-built executable if desired, while also allowing it to Just Work.
(this again touches on the problem of how there's several cases where py-binaries want to be used as part of the toolchain)
Probably, yes. CHECKED_HASH is somewhat appealing, too, though, because it allows you to modify the source file and see the new behavior without having to rebuild, though.
I'm +1 on defaulting to one and having an option for the other. Maybe base the default on
-c optso optimized builds used unchecked_hash by default, while fastbuilds use CHECKED_HASH by default. I can easily see a server not caring about this difference, while a CLI app would.I also like the idea of a library-level control here somehow, too. A scenario we've enountered with large packages (e.g. tensorflow), is the sheer number of modules has its own overhead; pyc helps, but being as how modifying them is the exception, not the rule, making them check their hash is wasted work. We'll have to figure out how a target-level and global-flag-level option should interact, though (which has precedence?).
Yeah, the details of the pyc infix aren't well understood to me. I think PEP 488 is another half of the puzzle. I swear I thought there was a part of the specification that included the "compile mode" of Python itself, too (e.g. debug for non-debug), but I don't see that mentioned.
We also have to make sure that the generated name is recognized by the runtime that will be used. I think this is controlled by BYTECODE_SUFFIXES? One option is to simply stick this onto the toolchain, too. Another would be to pass the necessary suffix to the stub template so it can configure the runtime at startup.
Additionally, the output path depends on if the original .py files is included or not.
The Major.Minor Python version is in the
--python_versionflag.adam-azarchs commentedon Oct 7, 2022
Yeah, this is kind of a similar problem as the one faced for the coverage tool in bazelbuild/bazel#15590. However, there's an important difference:
py_compileis part of the python standard library. Given that, I don't really see a need to offer an option to customization of the compilation tool -bazelcould just hard-code running$interpreter -m py_compile, which as far as I can tell would work for all of CPython, PyPy, and MicroPython at least (though I've only tried with CPython).Absolutely! However, that's something you can handle with
select, so it doesn't really have to impact the design much.I think the options we'd want exposed look something like
CHECKED_HASHUNCHECKED_HASHpyc-onlyAs well as the mostly orthogonal choices for "optimization" level. (which I will continue to put in quotes. It's really too bad that there isn't an optimization level which strips out docstrings, which you don't need in a server context, but leaves in assertions, which might be a bit dangerous to strip out)
This is why I suggested pre-compilation should be an attribute on the
py_libraryrule, as well as possibly the ability to set a default at the toolchain level. For one thing, you might want to have some things be distributed as pyc-only, while others might want to include source. You might also want to be able to control the "optimization" level on a per-target basis. I think the safe approach there would be to just say local specification takes precedence over global specification - I can't really imagine a situation where someone would set an override on a library and we'd not want to trust that they had a good reason to do so. At a nitty-gritty API level, however, it does mean we have to be careful about the value of the attribute one sets for this; one might think thatcompile_mode = Nonewould be a default of "use the toolchain default," but that would catch someone by surprise if they (somewhat reasonably) read that as meaning "no compilation". Easy enough to handle - just make the default for the attribute be e.g."from_toolchain"or whatever.rickeylev commentedon Oct 10, 2022
I think invoking the interpreter directly like that is a workable intermediate solution. So I'm +1 on doing that for now. I'm fairly certain we need a custom program to ensure deterministic behavior and pass the necessary info, though. The cli of -m py_compile isn't sufficient.
However, I need to make a few architectural points about why such solutions make it hard or impossible to take full advantage of Bazel's processing model and features.
As some concrete examples:
When using embedded Python, there's no need for a standalone interpreter, so building one simply for precompilation is unnecessary work. (This is particularly salient to me because that's my primary way of building Python programs).
Precompilation is a near perfect fit for using persistent workers, which would eliminate runtime startup overhead. This requires special support by the executable invoked.
Precompilation must run in the exec configuration. This means the runtime itself may be built differently than the runtime actually used, may be run remotely, and/or run on an entirely different platform. So more steps have to be taken to ensure such details don't leak into the output. For example, the launcher script does several things to prevent system settings from affecting the runtime which have to be re-implemented.
Similarly, the exec configuration usually emphasizes time-to-build over runtime-efficiency. This makes sense for typical user build tools, but less so for something like precompilation; precompilation is well defined and it's rate of invocation scales with the size of the overall build, ie the larger your transitive closure, the more it runs, so you're probably going to benefit from the precompiler (and runtime) being built with optimizations enabled. Such optimizations can extend beyond simply e.g. pyc -O levels, and into C-level optimizations that affect the runtime itself; these optimizations are hard to apply during a build, but are easy to apply to to an prebuilt executable which is later run.
This isn't to say simply doing "$runtime -m py_compile" is bad or wrong, just that it leaves a lot sitting at the Bazel Buffet Table.
Srsly, right? This has bugged me for years! It's the sort of thing a custom precompiler executable could do, though. This also makes me think, if we exposed an optimization level, it'd probably be better done as a list of values instead of a simple number.
I agree. A case we have internally is that protocol buffers (which are handled by a special rule) are compiled by default. This is pretty akin to a library-level setting.
I agree.
adam-azarchs commentedon Oct 10, 2022
Worth noting mostly for the sake of code archeology that there's this comment in the code currently: https://github.com/bazelbuild/bazel/blob/6d72ca979f8cf582f53452d5f905346e7effb113/src/main/java/com/google/devtools/build/lib/rules/python/PythonSemantics.java#L69-L71
however as far as I can tell the implementation https://github.com/bazelbuild/bazel/blob/9d0163002ca63f4cdbaff9420380d72e2a6e38b3/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPythonSemantics.java#L117-L120
simply copy the sources unaltered. But for what it's worth I'm guessing that at some point in the past someone had an implementation, or at least and attempt at one, probably Google-internal-only, for at least some of this functionality.
antonio-arbo commentedon Feb 27, 2023
Thanks for filing & tracking!
It would be great to have this option, particularly since this seems to render x-machine remote cache in bazel quite less effective since each machine has its own timestamp of the pyc. So it would be much better to have a more stable and safer hash instead of just timestamps.
Do you have a suggestion of a workaround for the time-being?
BTW, for others looking into this. A simple
bazel clean -expungeand then rebuild will show many remote cache misses. Confirm it looking into the exec logs as in https://bazel.build/remote/cache-remote .Ryang20718 commentedon Feb 15, 2024
I would very much <3 to see this feature.
we have a lot of heavy imports :(
Some of our heavy hitter imports bog down both test & runtime (mainly test)
One of the pain points right now is that we use pytorch in our bazel monorepo. However, because we import it in so many of our libraries, it incurs a large import time overhead (3.8 seconds).
With pycache, it drops down to ~2.5 seconds.
Is the general idea to generate pyc files once when all external third party repos are fetched at analysis time and then to ingest these files as input in
py_library?Would love some pointers on where to start (even if it's just a hacky prototype 😅 )
rickeylev commentedon Feb 15, 2024
I'm going to transfer this issue to the rules_python repo because, as of rules_python 0.31.0, we've enabled the rules_python-based implementation of the rules by default for Bazel 7+. This basically means this feature won't land in the Bazel builtin rules, but would instead be in the rules_python implementation.
rickeylev commentedon Feb 15, 2024
Can do! I've had parts of this mentally sketched out, but haven't thought it through entirely. This is already partially wired into the rules_python implementation, too.
There's a stub function,
maybe_precompile()src that gets called. As output, it returns the new set of sources a library should use, which is either (a) the py files (no precompiling), or (b) both the py files and pyc files, or (c) just the pyc files.All this function really has to do is loop over the sources, run a tool to generate the pyc files, and return them. The important thing is that it generates deterministic output (i.e. no timestamp based pyc files). It'll probably need some flags to control behavior a bit, as discussed, but lets focus on a prototype first. Anyways, the core code of this function is pretty simple:
That's the first part, which is easy.
The second part is defining the tool that is run to generate the pyc. For the purposes of experimenting, you can add an implicit attribute on py_library pointing to some executable to run. The tricky part here is the executable it points to can't be a regular py_binary -- that would result in a circular dependency. For experimenting, define it how you like to work out details (e.g. a sh_binary that just calls the system python would suffice for testing purposes).
Within Google, we prebuild a binary and use an implicit attribute. A prebuilt binary is actually a good thing in this case, but not required. It allows neat things like building an binary with optimizations, having a generic binary that can generate byte code for any python version, building a cc_binary that embeds python, or heck, you could implement a byte code compiler in Rust or whatever if you really wanted to. I'm getting ahead of myself. Anyways.
It's probably best to just start with invoking something like
python precompile.py <src> <pycout>. That's going to be pretty close to how a non-bazel python works.Using an implicit attribute, however -- that won't work well in the Bazel world where the environment isn't as tightly controlled and where there are many more platform configurations to worry about.
In the open Bazel world, this precompiler tool needs to come from a toolchain. This is because we need toolchain resolution to find a tool that matches the target config (the target python version we need to generate byte code for) that can run on one of our available execution platforms. A basic implementation that re-used the same runtime would look something like this, I think:
(there's variations of the above that would also work, but thats the gist).
Ryang20718 commentedon Mar 2, 2024
Was prototyping a bit. I really appreciate all the pointers here!
I used the system python interpreter to prototype for now and noticed some weird behavior and perhaps I'm missing something fundamental here, but when bazel "compiles" these python files into pyc, sometimes, we run into the case where the sandbox only contains the directory
bazel-out. cding into the sandbox debug path, we see only bazel-out exists and no "external" directory exists forexternal/pip_grpcio/site-packages/grpc/framework/interfaces/face/utilities.pyDebugging this prototype, we see src.path for this situation is
external/pip_grpcio/site-packages/grpc/framework/interfaces/face/utilities.pyand pyc.path isbazel-out/k8-opt/bin/external/pip_grpcio/site-packages/grpc/framework/interfaces/face/utilities.cpython-310.pycIs there something fundamental that I'm missing?
Prototype:
20 remaining items
adam-azarchs commentedon Mar 21, 2024
A trickier case to handle where you'd want to use
UNCHECKED_HASHis unit tests, for which bazel should always be guaranteeing the.pycis up to date (assuming they're running in either sandbox or remote), so you might as well take that optimization.Ryang20718 commentedon Mar 25, 2024
Was doing some prototyping to pipeline this
pyc_modethrough to PyInfo and had a couple of questionsI notice that in areas where
maybe_precompileis used, PyInfo is generated with the outputs ofmaybe_precompile. I.e hereAssuming we'd want the end user to be able to do something like
Would we need to propagate from PY_SRCS_ATTRS? Or is there a way to do this solely via PyInfo?
Currently, I just use a
sh_binarywhich wraps a local python, but ideally it would match the python toolchain defined?Does bzlmod support pytest with multiple pip/python versions? I see there's this comment here
rickeylev commentedon Apr 16, 2024
Coincidentally, we have some internal work in this area planned, so I thought I'd post some of our analysis that seemed relevant.
Note that this isn't documenting a decision. It's merely trying to inform from looking at the use case we're trying to solve (in brief, precompiling enabled by default) and the constraints we have.
A
precompileattribute with 5 values seems very useful:It seems like there are 3 flags to decide behavior?
--precompile_default: used when a target doesn't have a value set (or when its inherit). It would have values:--precompile_force: Like--precompile_defaultbut overrides target level settings. Probably the same value as--precompile_default. (The use cases for this seem valid but niche, see below).--precompile_force_retain_sources: Forces that py sources are retained. e.g. precompile=only acts like precompile=all (mostly an alternative to precompile_force, see below)re: flag vs attribute precedence: this is a really tough question. We could come up with reasonable scenarios where both are sensible behaviors. Overall, the attribute taking precedence seems like the better default behavior. The main case for having the flag take precedence is: what if a target you don't own sets precompile=only (having just pyc makes sense if it just generates code), but you have to deploy to an environment where that pyc won't be valid (i.e. your build is for Python 3.10, but you deploy to something with python 3.9). This is somewhat of an edge case for us, but seems like it would be much more common in OSS. This case is also where the
--precompile_force_retain_sourcesflag comes from: by forcing sources to be kept, the invalid pyc wouldn't be an issue in practice.The above case also implies that there's a distinction between binaries and libraries in this regard (i.e. a binary level-setting applies transitively). As a general rule, if there's a global flag to set the default or force the state, then it probably should also be a binary-level attribute. It's not always possible or desirable to set a global flag, especially if there's just a handful of targets that need it.
That said, a binary level setting also comes with problems. As an example, given:
Then the problem is the library can't directly see what the binary needs, and only outputs
foo.pyc, when it should have output bothfoo.pyand__pycache__/foo.pyc. We think an aspect can address this, but its illustrative of some of the caveats and complex interactions that can happen (and also how transitions make it easy for an unfortunately high cost).rickeylev commentedon Apr 16, 2024
I'm not sure I understand the question. What goes into PyInfo is derived from the target attributes (srcs and deps, usually). The core logic, when unrolled, should basically be:
Does that answer your question?
I think you're hinting at the issue of you need an exec-config interpreter, but the python toolchain only has a target-config interpreter?
I think your 3 basic options are:
I think a combination of (2) and (3) is probably the best? (2) invokes toolchain resolution, which permits the target config to influence selection (i.e. allows the target config pythonversion=3.10 to select a toolchain that can generate 3.10 bytecode). (3) allows abstracting how the precompiler binary is implemented (i.e. it could be a pre-compiled binary or simply lookup the "regular" interpreter).
Yes, but recall how toolchain resolution and matching works: it's taking the key:value constraints and looking over lists to find something that matches. I say this because it sounds like you might be thinking/desiring this:
regular_toolchain = match(X, "regular"); precompile_toolchain = create_precompile_toolchain_from(regular_toolchain), but that isn't how toolchains operate. It's more like this:regular_toolchain = match(X, "regular"); precompile_toolchain = match(X, "precompile").Yes, it supports multiple versions. That comment is out of date.
adam-azarchs commentedon Apr 18, 2024
This can be avoided if we're willing to complicate the
PyInfoto separately track sets of files for each use case as I suggested in #1761 (comment). This is probably not a great approach, however, as there are probably a lot of existing rules out there that propagatePyInfoand might start silently dropping these additional fields.rickeylev commentedon May 12, 2024
I've been working on adding this functionality. You can see the WIP here: main...rickeylev:rules_python:precompile.toolchain
Everything appears to work, so I've started cleaning it up.
The basic gist of what I have implemented is:
persistent worker. (I started on this path for fun, but then it turned out to be a
huge performance benefit -- 60+ seconds vs 4 seconds when building the 2000 file
bzlmod example)
precompile=inherit|enabled|disabled|if_generated_source: controls if a targetperforms precompilation.
--precompile=auto|enabled|disabled|if_generated_source|force_enabled|force_disabledprecompile_invalidation_mode=auto|checked_hash|unchecked_hash: determines what invalidation mode to use.precompile_optimize_level=<int>: Controls the optimization level.--precompile_source_retention=auto|keep_source|omit_source|omit_if_generated_source: controls if the py is included in the output or not.pyc_collection: controls if binaries explicitly include pyc filesfrom dependencies. This derives from the Google situation, where controlling behavior
at the binary level makes it easier to incrementally roll out the change.
--precompile_add_to_runfiles=auto|always|decided_elsewhere: controls if the pyc filesare added to the runfiles of the generating target (e.g. if py_library adds its pyc
files to the runfiles it returns). This is used in combination with
pyc_collection.The two basic ways to use precompilation are:
can then opt-into including the pyc files. This is more for the Google situation, but
really any situation where enabling precompiling at large doesn't Just Work.
If you like optimizing Python code, then optimizing the precompiler tool might be fun. Unfortunately, precompiling adds a decent amount of overhead. As an example, the bzlmod example has about 2,000 py files. This takes about 60+ seconds to precompile as a regular action (1 py file per action) 🙁.
So I implemented a second precompiler that is a multiplexed, sandbox-aware, persistent worker. This reduced build time from 60s to about 4 seconds (yes, single-digit). I think we can do better, but I haven't explored where the bottleneck is. If we can change it to be a prebuilt binary, then we'd have more options for optimizing it, too.
adam-azarchs commentedon May 14, 2024
A few observations:
Persistent worker
This approach worries me a bit because there's a lot of rough edges around using a persistent worker with remote execution (which my team uses exclusively).
Python interpreter startup time is a lot; it's not surprising that you get significant benefits from a persistent worker, especially since AFAKT the implementation for
py_compileis to actually fully import the module, which ends up processing all of its transitive imports etc. Could probably improve on that very significantly with a purpose-built binary. I haven't actually looked into how hard that would be to write, but at first glance at least if you're allowed to linklibpythonit looks like you'd need toPyImport_GetMagicNumberfor the header_imp_source_hashis the right call to make here but that's a private symbol. You could go through the python extension module_imp.source_hashbut at that point I'm not sure how much benefit from avoiding a full interpreter is preserved...PyMarshal_WriteObjectToStringto serialize itpyc-only mode
I'm not sure this works in if
precompile="disabled"is set on any dependencies. I think you can get around that by adding the sources into the "pyc" depset in that situation, though that muddies the meaning a bit.You're also not supporting the use case of permitting specific libraries to be pyc-only without setting it overall. This could be done; similarly to precompile=disabled you'd need to add the pyc to the normal sources as well as as pycs in order to support cases that were not collecting pyc.
rickeylev commentedon May 14, 2024
re: persistent worker: Yes, all good points. The precompiler works as a non-worker, regular worker, or multiplexed worker, and there's a flag to set the execution_requirements for the precompile action.
re: py_compile fully imports: It doesn't import (execute) the py code. It just uses the importlib libraries to read the file and compile it to pyc. I found that surprising, but maybe not so much if this is closely coupled to the import system's processing of things (e.g.
_imp_source_hashbeing part of import.c).re: linking with libpython: Yes, that'd be awesome. I'd love to get a cc_binary that statically links with libpython working. For something like the precompiler, which is a single file that only uses the stdlib, such a thing seems doable.
re: library-level pyc-only: Ah yeah, I forgot about that. That should be doable with target-level attribute to control that.
adam-azarchs commentedon May 14, 2024
I forgot that you can't actually choose to be pyc-only at the
py_binarylevel because the file naming requirements are different for pyc-only vs. py+pyc. So that decision needs to be done at thepy_librarylevel, but can have a global default at least.I'll maybe take a time-boxed crack at trying to get such a
cc_binaryworking tomorrow.rickeylev commentedon May 15, 2024
re: remote builds, performance, persistent workers: A thought: Right now, each source file is processed with a separate action. Some performance could be regained with non-worker execution by sending all of a target's sources to a single invocation. This would at least change the overhead to once-per-target instead of once-per-file. For targets with many sources (e.g. something from pypi), this would probably help a decent amount.
adam-azarchs commentedon May 15, 2024
One of our repos using remote execution:
adam-azarchs commentedon May 15, 2024
Didn't really have time to figure out a truly portable way to do the source hashing, but I did try a different, much simpler python implementation:
This avoids
argparseimporting of all of the stuff you need for a persistent worker, and seems to reliably take less than half the time as yourprecompile.pywhen invoked on a single file. This isn't that surprising; whatargparseis doing is pretty expensive if you're going to be doing it a huge number of times, though it's worth it for a longer-lived process.This does suggest that the persistent worker implementation should at the very least be kept separate from the single-file implementation. Or at least should process imports lazily as-needed.
rickeylev commentedon May 16, 2024
Oh, nice analysis. I see similar. All those unused imports add up.
re: lazy imports: Yeah, I'll do this, at the least. This is an easy change to make.
re: splitting worker impl into another file: That's a good idea, too; it would avoid having to parse all the worker-related code. It'll have to wait for another PR, though.
The argparse import itself isn't too bad, so I'm going to keep the
--flagbased calling convention. Actually, its more an@fileconvention with the args within in. We don't know if the action will use a worker or not, so the args have to always be passed to the param file.feat: compile source files at build time (#1902)