diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d4f487a030a8..d01af76edf0e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,87 @@ ## Next Release +## Mypy 2.1 + +We’ve just uploaded mypy 2.1.0 to the Python Package Index ([PyPI](https://pypi.org/project/mypy/)). +Mypy is a static type checker for Python. This release includes new features, performance +improvements and bug fixes. You can install it as follows: + + python3 -m pip install -U mypy + +You can read the full documentation for this release on [Read the Docs](http://mypy.readthedocs.io). + +### librt.vecs: Fast Growable Array Type for Mypyc + +The new `librt.vecs` module provides an efficient growable array type `vec` that is +optimized for mypyc use. It provides fast, packed arrays with integer and floating point +value types, which can be **several times faster** than `list`, and tens of times faster +than `array.array` in code compiled using mypyc. It also supports nested `vec` objects and +non-value-type items, such as ``vec[vec[str]]``. + +Refer to the [documentation](https://mypyc.readthedocs.io/en/latest/librt_vecs.html) for +the details. + +Contributed by Jukka Lehtosalo. + +### librt.random: Fast Pseudo-Random Number Generation + +The new `librt.random` module provides fast pseudo-random number generation that is +optimized for code compiled using mypyc. It can be 3x to 10x faster than the stdlib +`random` module in compiled code. + +Refer to the [documentation](https://mypyc.readthedocs.io/en/latest/librt_random.html) for +the details. + +Contributed by Jukka Lehtosalo (PR [21433](https://github.com/python/mypy/pull/21433)). + +### Mypyc Improvements + +- Make compilation order with multiple files consistent (Piotr Sawicki, PR [21419](https://github.com/python/mypy/pull/21419)) +- Fix crash on accessing `StopAsyncIteration` (Piotr Sawicki, PR [21406](https://github.com/python/mypy/pull/21406)) +- Fix incremental compilation with `separate` flag (Vaggelis Danias, PR [21299](https://github.com/python/mypy/pull/21299)) + +### Fixes to Crashes + +- Fix crash on partial type with `--allow-redefinition` and `global` declaration (Jukka Lehtosalo, PR [21428](https://github.com/python/mypy/pull/21428)) +- Fix broken awaitable generator patching (Ivan Levkivskyi, PR [21435](https://github.com/python/mypy/pull/21435)) + +### Changes to Messages + +- Fix function call error message for small number of arguments (sobolevn, PR [21432](https://github.com/python/mypy/pull/21432)) + +### Other Notable Fixes and Improvements + +- Rely on typeshed stubs for `slice` typing (Ivan Levkivskyi, PR [21401](https://github.com/python/mypy/pull/21401)) +- Improve negative narrowing for membership checks on tuples (Shantanu, PR [21456](https://github.com/python/mypy/pull/21456)) +- Narrow match captures based on previous cases (Shantanu, PR [21405](https://github.com/python/mypy/pull/21405)) +- Fix nondeterminism in overload resolution (Shantanu, PR [21455](https://github.com/python/mypy/pull/21455)) +- Respect file config comments for stale modules (Adam Turner, PR [21444](https://github.com/python/mypy/pull/21444)) +- Fix JSON output mode for syntax errors in parallel mode (Adam Turner, PR [21434](https://github.com/python/mypy/pull/21434)) +- Fix type variable with values as a supertype (Ivan Levkivskyi, PR [21431](https://github.com/python/mypy/pull/21431)) +- Add support for configuring `--num-workers` with an environment variable (Kevin Kannammalil, PR [21407](https://github.com/python/mypy/pull/21407)) +- Respect JSON output mode for syntax errors (Adam Turner, PR [21386](https://github.com/python/mypy/pull/21386)) +- Analyze `TypedDict` decorators (Pranav Manglik, PR [21267](https://github.com/python/mypy/pull/21267)) + +### Typeshed Updates + +Please see [git log](https://github.com/python/typeshed/commits/main?after=e4d32e01bee44241a5e7c33298c261175b9f1bdb+0&branch=main&path=stdlib) for full list of standard library typeshed stub changes. + +### Acknowledgements + +Thanks to all mypy contributors who contributed to this release: + +- Adam Turner +- Ivan Levkivskyi +- Jukka Lehtosalo +- Kevin Kannammalil +- Piotr Sawicki +- Shantanu +- sobolevn +- Vaggelis Danias + +I’d also like to thank my employer, Dropbox, for supporting mypy development. + ## Mypy 2.0 We’ve just uploaded mypy 2.0.0 to the Python Package Index ([PyPI](https://pypi.org/project/mypy/)). diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index b8e4c7c187ba4..d44c08fa48d31 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -1211,7 +1211,7 @@ format into the specified directory. Enabling incomplete/experimental features ***************************************** -.. option:: --enable-incomplete-feature {PreciseTupleTypes,InlineTypedDict} +.. option:: --enable-incomplete-feature {PreciseTupleTypes,InlineTypedDict,TypeForm} Some features may require several mypy releases to implement, for example due to their complexity, potential for backwards incompatibility, or @@ -1266,6 +1266,8 @@ List of currently incomplete/experimental features: def test_values() -> {"width": int, "description": str}: return {"width": 42, "description": "test"} +* ``TypeForm``: this feature enables ``TypeForm``, as described in + `PEP 747 – Annotating Type Forms _`. Miscellaneous diff --git a/mypy-requirements.txt b/mypy-requirements.txt index 27c76a0f3f6a8..0216f47852baa 100644 --- a/mypy-requirements.txt +++ b/mypy-requirements.txt @@ -5,5 +5,5 @@ typing_extensions>=4.14.0; python_version>='3.15' mypy_extensions>=1.0.0 pathspec>=1.0.0 tomli>=1.1.0; python_version<'3.11' -librt>=0.10.0; platform_python_implementation != 'PyPy' +librt>=0.11.0; platform_python_implementation != 'PyPy' ast-serialize>=0.3.0,<1.0.0 diff --git a/mypy/build.py b/mypy/build.py index 21a5559b329a3..8d5db0bab8dfa 100644 --- a/mypy/build.py +++ b/mypy/build.py @@ -4769,13 +4769,13 @@ def process_stale_scc(graph: Graph, ascc: SCC, manager: BuildManager) -> None: t2 = time.time() stale = scc + # Parse before verify_dependencies so that inline config comments + # (e.g. "# mypy: disable-error-code") are applied to options. + manager.parse_all([graph[id] for id in stale], post_parse=False) for id in stale: # Re-generate import errors in case this module was loaded from the cache. if graph[id].meta: graph[id].verify_dependencies(suppressed_only=True) - # We may already have parsed the modules, or not. - # If the former, parse_file() is a no-op. - manager.parse_all([graph[id] for id in stale], post_parse=False) if "typing" in scc: # For historical reasons we need to manually add typing aliases # for built-in generic collections, see docstring of diff --git a/mypy/checkexpr.py b/mypy/checkexpr.py index 123c5f821ed29..71ffa7c4ff23b 100644 --- a/mypy/checkexpr.py +++ b/mypy/checkexpr.py @@ -1787,6 +1787,7 @@ def check_callable_call( might_have_shifted_args = ( not self.msg.prefer_simple_messages() + and len(args) >= 2 # see gh-21427 and all(k == ARG_POS for k in callee.arg_kinds) and all(k == ARG_POS for k in arg_kinds) and len(arg_kinds) == len(callee.arg_kinds) - 1 diff --git a/mypy/options.py b/mypy/options.py index 2f03cd3eab5b7..4281c88e67dea 100644 --- a/mypy/options.py +++ b/mypy/options.py @@ -94,8 +94,8 @@ class BuildType: NEW_GENERIC_SYNTAX: Final = "NewGenericSyntax" INLINE_TYPEDDICT: Final = "InlineTypedDict" TYPE_FORM: Final = "TypeForm" -INCOMPLETE_FEATURES: Final = frozenset((PRECISE_TUPLE_TYPES, INLINE_TYPEDDICT)) -COMPLETE_FEATURES: Final = frozenset((TYPE_VAR_TUPLE, UNPACK, NEW_GENERIC_SYNTAX, TYPE_FORM)) +INCOMPLETE_FEATURES: Final = frozenset((PRECISE_TUPLE_TYPES, INLINE_TYPEDDICT, TYPE_FORM)) +COMPLETE_FEATURES: Final = frozenset((TYPE_VAR_TUPLE, UNPACK, NEW_GENERIC_SYNTAX)) class Options: diff --git a/mypy/semanal.py b/mypy/semanal.py index a958043fa35c2..f9b52a0dcfba8 100644 --- a/mypy/semanal.py +++ b/mypy/semanal.py @@ -195,7 +195,7 @@ type_aliases_source_versions, typing_extensions_aliases, ) -from mypy.options import Options +from mypy.options import TYPE_FORM, Options from mypy.patterns import ( AsPattern, ClassPattern, @@ -3701,7 +3701,8 @@ def analyze_lvalues(self, s: AssignmentStmt) -> None: ) def analyze_rvalue_as_type_form(self, s: AssignmentStmt) -> None: - self.try_parse_as_type_expression(s.rvalue) + if TYPE_FORM in self.options.enable_incomplete_feature: + self.try_parse_as_type_expression(s.rvalue) def apply_dynamic_class_hook(self, s: AssignmentStmt) -> None: if not isinstance(s.rvalue, CallExpr): @@ -5451,7 +5452,8 @@ def visit_return_stmt(self, s: ReturnStmt) -> None: self.fail('"return" not allowed in except* block', s, serious=True) if s.expr: s.expr.accept(self) - self.try_parse_as_type_expression(s.expr) + if TYPE_FORM in self.options.enable_incomplete_feature: + self.try_parse_as_type_expression(s.expr) self.statement = old def visit_raise_stmt(self, s: RaiseStmt) -> None: @@ -6054,9 +6056,11 @@ def visit_call_expr(self, expr: CallExpr) -> None: expr.analyzed.accept(self) else: # Normal call expression. + calculate_type_forms = TYPE_FORM in self.options.enable_incomplete_feature for a in expr.args: a.accept(self) - self.try_parse_as_type_expression(a) + if calculate_type_forms: + self.try_parse_as_type_expression(a) if ( isinstance(expr.callee, MemberExpr) diff --git a/mypy/solve.py b/mypy/solve.py index e3709106996cd..4a5eec47ca60d 100644 --- a/mypy/solve.py +++ b/mypy/solve.py @@ -17,6 +17,7 @@ AnyType, Instance, NoneType, + Overloaded, ParamSpecType, ProperType, TupleType, @@ -253,6 +254,9 @@ def _join_sorted_key(t: Type) -> int: return -2 if isinstance(t, NoneType): return -1 + + if isinstance(t, Overloaded): + return 1 return 0 diff --git a/mypy/typeanal.py b/mypy/typeanal.py index db56256192625..3351bc1a2ca61 100644 --- a/mypy/typeanal.py +++ b/mypy/typeanal.py @@ -49,7 +49,7 @@ check_arg_kinds, check_param_names, ) -from mypy.options import INLINE_TYPEDDICT, Options +from mypy.options import INLINE_TYPEDDICT, TYPE_FORM, Options from mypy.plugin import AnalyzeTypeContext, Plugin, TypeAnalyzerPluginInterface from mypy.semanal_shared import ( SemanticAnalyzerCoreInterface, @@ -674,6 +674,12 @@ def try_analyze_special_unbound_type(self, t: UnboundType, fullname: str) -> Typ item = AnyType(TypeOfAny.from_error) return TypeType.make_normalized(item, line=t.line, column=t.column) elif fullname in ("typing_extensions.TypeForm", "typing.TypeForm"): + if TYPE_FORM not in self.options.enable_incomplete_feature: + self.fail( + "TypeForm is experimental," + " must be enabled with --enable-incomplete-feature=TypeForm", + t, + ) if len(t.args) == 0: any_type = self.get_omitted_any(t) return TypeType(any_type, line=t.line, column=t.column, is_type_form=True) diff --git a/mypy/typeshed/stubs/librt/librt/random.pyi b/mypy/typeshed/stubs/librt/librt/random.pyi new file mode 100644 index 0000000000000..d1330aa56faf1 --- /dev/null +++ b/mypy/typeshed/stubs/librt/librt/random.pyi @@ -0,0 +1,22 @@ +from typing import final, overload + +from mypy_extensions import i64 + +def random() -> float: ... +def randint(a: i64, b: i64) -> i64: ... +@overload +def randrange(stop: i64, /) -> i64: ... +@overload +def randrange(start: i64, stop: i64, /) -> i64: ... +def seed(n: i64, /) -> None: ... + +@final +class Random: + def __init__(self, seed: i64 | None = None) -> None: ... + def randint(self, a: i64, b: i64) -> i64: ... + @overload + def randrange(self, stop: i64, /) -> i64: ... + @overload + def randrange(self, start: i64, stop: i64, /) -> i64: ... + def random(self) -> float: ... + def seed(self, n: i64, /) -> None: ... diff --git a/mypy/version.py b/mypy/version.py index 82a0d52db14f7..9f73dd736f677 100644 --- a/mypy/version.py +++ b/mypy/version.py @@ -8,7 +8,7 @@ # - Release versions have the form "1.2.3". # - Dev versions have the form "1.2.3+dev" (PLUS sign to conform to PEP 440). # - Before 1.0 we had the form "0.NNN". -__version__ = "2.1.0+dev" +__version__ = "2.1.0" base_version = __version__ mypy_dir = os.path.abspath(os.path.dirname(os.path.dirname(__file__))) diff --git a/mypyc/build.py b/mypyc/build.py index d55334d8ac800..84633086d2724 100644 --- a/mypyc/build.py +++ b/mypyc/build.py @@ -121,6 +121,7 @@ class ModDesc(NamedTuple): ["vecs"], ), ModDesc("librt.time", ["time/librt_time.c"], ["time/librt_time.h"], []), + ModDesc("librt.random", ["random/librt_random.c"], ["random/librt_random.h"], ["random"]), ] try: @@ -449,6 +450,70 @@ def write_file(path: str, contents: str) -> None: os.utime(path, times=(new_mtime, new_mtime)) +_MYPYC_EXTENSION_MARKER = "_mypyc_skip_redundant_inplace_copy" +_setuptools_patch_applied = False + + +def _patch_setuptools_copy_extensions_to_source() -> None: + """Skip redundant `.so` copies for extensions we generated. + + setuptools' copy_extensions_to_source rewrites every `.so` in the + source tree on every build_ext, even when nothing changed. On macOS + this invalidates AMFI's signature cache (~100 ms re-verification per + `.so` on the next import), eating most of the separate=True + incremental speedup. + + The patch is global because copy_extensions_to_source runs during + setup()'s build_ext command, after mypycify() has already returned; + we can't scope a context manager around it. Instead the skip only + fires for extensions tagged by mypycify (via the marker attribute), + so other setuptools users in the same setup.py see the unmodified + upstream behavior, including stub writes. + """ + global _setuptools_patch_applied + if _setuptools_patch_applied: + return + _setuptools_patch_applied = True + + from setuptools.command.build_ext import build_ext as _build_ext + + original = _build_ext.copy_extensions_to_source + + def _files_match(a: str, b: str) -> bool: + try: + sa = os.stat(a) + sb = os.stat(b) + except OSError: + return False + # Compare size + whole-second mtime. distutils' copy_file + # propagates the source mtime, but macOS drops sub-second + # precision on write so the float values never match verbatim. + return sa.st_size == sb.st_size and int(sa.st_mtime) == int(sb.st_mtime) + + def patched(self: Any) -> None: + build_py = self.get_finalized_command("build_py") + + def is_redundant(ext: Any) -> bool: + if not getattr(ext, _MYPYC_EXTENSION_MARKER, False): + return False + inplace_file, regular_file = self._get_inplace_equivalent(build_py, ext) + return _files_match(regular_file, inplace_file) + + # Hide our already-fresh extensions from setuptools' loop and + # let it handle whatever's left. Delegating instead of + # reimplementing the body means future setuptools changes carry + # over for free. self.extensions is restored before we return + # so anything that inspects it later sees the original list. + saved = self.extensions + self.extensions = [ext for ext in saved if not is_redundant(ext)] + try: + original(self) + finally: + self.extensions = saved + + _build_ext.copy_extensions_to_source = patched # type: ignore[method-assign] + + def construct_groups( sources: list[BuildSource], separate: bool | list[tuple[list[str], str | None]], @@ -512,7 +577,7 @@ def get_header_deps(cfiles: list[tuple[str, str]]) -> list[str]: """ headers: set[str] = set() for _, contents in cfiles: - headers.update(re.findall(r'#include "(.*)"', contents)) + headers.update(re.findall(r'#include [<"]([^>"]+)[>"]', contents)) return sorted(headers) @@ -572,12 +637,21 @@ def mypyc_build( cfilenames = [] for cfile, ctext in cfiles: cfile = os.path.join(compiler_options.target_dir, cfile) - if not options.mypyc_skip_c_generation: + # Empty contents marks a file the previous run already wrote + # (fully-cached group): skip the rewrite and just reuse it. + if ctext and not options.mypyc_skip_c_generation: write_file(cfile, ctext) if os.path.splitext(cfile)[1] == ".c": cfilenames.append(cfile) - deps = [os.path.join(compiler_options.target_dir, dep) for dep in get_header_deps(cfiles)] + # The header regex matches both quote styles, so the result can + # include system headers like `` that don't live under + # target_dir. Joining those produces non-existent paths which + # would force a full rebuild on every run via Extension.depends. + candidate_deps = ( + os.path.join(compiler_options.target_dir, dep) for dep in get_header_deps(cfiles) + ) + deps = [d for d in candidate_deps if os.path.exists(d)] group_cfilenames.append((cfilenames, deps)) return groups, group_cfilenames, source_deps @@ -631,6 +705,9 @@ def get_cflags( # Disables C Preprocessor (cpp) warnings # See https://github.com/mypyc/mypyc/issues/956 "-Wno-cpp", + "-Wno-array-bounds", + "-Wno-stringop-overread", + "-Wno-stringop-overflow", ] if log_trace: cflags.append("-DMYPYC_LOG_TRACE") @@ -751,6 +828,9 @@ def mypycify( have no backward compatibility guarantees! """ + # Skip redundant inplace .so copies on every build_ext invocation. + _patch_setuptools_copy_extensions_to_source() + # Figure out our configuration compiler_options = CompilerOptions( strip_asserts=strip_asserts, @@ -865,4 +945,9 @@ def mypycify( ) ) + # Tag every extension we own so the build_ext patch knows it's + # safe to skip the redundant inplace copy for these specifically. + for ext in extensions: + setattr(ext, _MYPYC_EXTENSION_MARKER, True) + return extensions diff --git a/mypyc/codegen/emit.py b/mypyc/codegen/emit.py index 45ff34ab045d5..01cf3593a8d60 100644 --- a/mypyc/codegen/emit.py +++ b/mypyc/codegen/emit.py @@ -326,6 +326,18 @@ def get_group_prefix(self, obj: ClassIR | FuncDecl) -> str: # See docs above return self.get_module_group_prefix(obj.module_name) + def register_group_dep(self, cl: ClassIR) -> None: + """Record `cl`'s defining group as a cross-group dep, if any. + + Call this when emitting code that refers to `cl`'s struct + layout: the .c file consuming that layout needs the defining + group's `__native_*.h` included, and group_deps drives which + headers get pulled in. + """ + target_group = self.context.group_map.get(cl.module_name) + if target_group and target_group != self.context.group_name: + self.context.group_deps.add(target_group) + def static_name(self, id: str, module: str | None, prefix: str = STATIC_PREFIX) -> str: """Create name of a C static variable. diff --git a/mypyc/codegen/emitfunc.py b/mypyc/codegen/emitfunc.py index e4a8922a103d4..dcb606f6ab51b 100644 --- a/mypyc/codegen/emitfunc.py +++ b/mypyc/codegen/emitfunc.py @@ -360,6 +360,11 @@ def get_attr_expr(self, obj: str, op: GetAttr | SetAttr, decl_cl: ClassIR) -> st classes, and *(obj + attr_offset) for attributes defined by traits. We also insert all necessary C casts here. """ + # The struct cast below needs the defining group's __native.h + # included by the consuming .c file. Record both the receiver + # and declaring classes as cross-group deps. + self.emitter.register_group_dep(op.class_type.class_ir) + self.emitter.register_group_dep(decl_cl) cast = f"({op.class_type.struct_name(self.emitter.names)} *)" if decl_cl.is_trait and op.class_type.class_ir.is_trait: # For pure trait access find the offset first, offsets diff --git a/mypyc/codegen/emitmodule.py b/mypyc/codegen/emitmodule.py index 2025426188412..fa0a4385f4fb5 100644 --- a/mypyc/codegen/emitmodule.py +++ b/mypyc/codegen/emitmodule.py @@ -59,6 +59,7 @@ from mypyc.errors import Errors from mypyc.ir.deps import ( LIBRT_BASE64, + LIBRT_RANDOM, LIBRT_STRINGS, LIBRT_TIME, LIBRT_VECS, @@ -362,7 +363,12 @@ def compile_ir_to_c( if source.module in modules } if not group_modules: - ctext[group_name] = [] + # Fully-cached group (e.g. pip's second setup.py invoke for + # the wheel phase): no fresh IR was produced. Reuse the file + # list recorded in any module's IR cache so the linker still + # sees the previous run's outputs; empty content is a "do + # not rewrite" sentinel for mypyc_build. + ctext[group_name] = _load_cached_group_files(group_sources, result) continue generator = GroupGenerator( group_modules, source_paths, group_name, mapper.group_map, names, compiler_options @@ -372,6 +378,32 @@ def compile_ir_to_c( return ctext +def _load_cached_group_files( + group_sources: list[BuildSource], result: BuildResult +) -> list[tuple[str, str]]: + """Read the .c/.h paths recorded for this group on the previous run. + + All modules in a group share the same src_hashes map, so the first + readable IR cache is sufficient. Returns paths paired with empty + content so callers can distinguish "reuse on disk" from "newly + generated". + """ + for source in group_sources: + state = result.graph.get(source.module) + if state is None: + continue + try: + ir_json = result.manager.metastore.read(get_state_ir_cache_name(state)) + except (FileNotFoundError, OSError): + continue + try: + ir_data = json.loads(ir_json) + except json.JSONDecodeError: + continue + return [(path, "") for path in ir_data.get("src_hashes", {})] + return [] + + def get_ir_cache_name(id: str, path: str, options: Options) -> str: meta_path, _, _ = get_cache_names(id, path, options) # Mypyc uses JSON cache even with --fixed-format-cache (for now). @@ -614,16 +646,19 @@ def generate_c_for_modules(self) -> list[tuple[str, str]]: base_emitter = Emitter(self.context) # Optionally just include the runtime library c files to - # reduce the number of compiler invocations needed + # reduce the number of compiler invocations needed. + # Use <> form (only -I paths) so a shim file with the same + # basename as a runtime file can't shadow it. Triggered by + # mypyc/lower/int_ops.py vs lib-rt/int_ops.c on mypy self-compile. if self.compiler_options.include_runtime_files: for name in RUNTIME_C_FILES: - base_emitter.emit_line(f'#include "{name}"') + base_emitter.emit_line(f"#include <{name}>") # Include conditional source files source_deps = collect_source_dependencies(self.modules) for source_dep in sorted(source_deps, key=lambda d: d.path): - base_emitter.emit_line(f'#include "{source_dep.path}"') + base_emitter.emit_line(f"#include <{source_dep.path}>") if self.compiler_options.depends_on_librt_internal: - base_emitter.emit_line('#include "internal/librt_internal_api.c"') + base_emitter.emit_line("#include ") base_emitter.emit_line(f'#include "__native{self.short_group_suffix}.h"') base_emitter.emit_line(f'#include "__native_internal{self.short_group_suffix}.h"') emitter = base_emitter @@ -1224,6 +1259,10 @@ def emit_module_exec_func( emitter.emit_line("if (import_librt_vecs() < 0) {") emitter.emit_line("return -1;") emitter.emit_line("}") + if LIBRT_RANDOM in module.dependencies: + emitter.emit_line("if (import_librt_random() < 0) {") + emitter.emit_line("return -1;") + emitter.emit_line("}") emitter.emit_line("PyObject* modname = NULL;") if self.multi_phase_init: emitter.emit_line(f"{module_static} = module;") diff --git a/mypyc/doc/index.rst b/mypyc/doc/index.rst index 8be1086bdd707..aacf275de9885 100644 --- a/mypyc/doc/index.rst +++ b/mypyc/doc/index.rst @@ -33,6 +33,7 @@ generate fast code. librt librt_base64 + librt_random librt_strings librt_time librt_vecs diff --git a/mypyc/doc/librt.rst b/mypyc/doc/librt.rst index a9492e1d61268..23206f8cfe806 100644 --- a/mypyc/doc/librt.rst +++ b/mypyc/doc/librt.rst @@ -26,6 +26,8 @@ Follow submodule links in the table to a detailed description of each submodule. - Description * - :doc:`librt.base64 ` - Fast Base64 encoding and decoding + * - :doc:`librt.random ` + - Pseudorandom number generation * - :doc:`librt.strings ` - String and bytes utilities * - :doc:`librt.time ` diff --git a/mypyc/doc/librt_random.rst b/mypyc/doc/librt_random.rst new file mode 100644 index 0000000000000..d5543661ce987 --- /dev/null +++ b/mypyc/doc/librt_random.rst @@ -0,0 +1,97 @@ +.. _librt-random: + +librt.random +============ + +The ``librt.random`` module is part of the ``librt`` package on PyPI, and it provides +pseudorandom number generation utilities. It can be used as a significantly faster +alternative to the stdlib :mod:`random` module in compiled code. It can also be faster +than stdlib ``random`` in interpreted code, depending on use case. + +The module uses the `ChaCha8 `__ algorithm with forward +secrecy. It is **not** suitable for cryptographic use, but it provides high-quality, +statistically uniform output. + +Functions +--------- + +The module provides module-level functions that use thread-local state, so they are +safe to call concurrently from multiple threads without external locking, and they +scale well even if used from multiple threads: + +.. function:: random() -> float + + Return a random floating-point number in the range [0.0, 1.0). + +.. function:: randint(a: i64, b: i64) -> i64 + + Return a random integer *n* such that *a* <= *n* <= *b*. + +.. function:: randrange(stop: i64, /) -> i64 + randrange(start: i64, stop: i64, /) -> i64 + + Return a random integer from the range. With one argument, the range is [0, *stop*). + With two arguments, the range is [*start*, *stop*). + +.. function:: seed(n: i64, /) -> None + + Seed the thread-local random number generator. This only affects module-level + functions called from the current thread. + +Random class +------------ + +.. class:: Random(seed: i64 | None = None) + + A pseudorandom number generator instance with its own independent state. Use this + when you need reproducible sequences or want to avoid interference with the + thread-local state used by the module-level functions. + + If *seed* is ``None``, the generator is seeded from OS entropy + (via :func:`os.urandom`). + + It's not safe to use the same ``Random`` instance concurrently from multiple + threads without synchronization on free-threaded Python builds. + + .. method:: random() -> float + + Return a random floating-point number in the range [0.0, 1.0). + + .. method:: randint(a: i64, b: i64) -> i64 + + Return a random integer *n* such that *a* <= *n* <= *b*. + + .. method:: randrange(stop: i64, /) -> i64 + randrange(start: i64, stop: i64, /) -> i64 + + Return a random integer from the range. With one argument, the range is [0, *stop*). + With two arguments, the range is [*start*, *stop*). + + .. method:: seed(n: i64, /) -> None + + Reseed the generator. + +Example +------- + +Using module-level functions:: + + from librt.random import randint, seed + + def roll_dice() -> i64: + return randint(1, 6) + +Using a ``Random`` instance for reproducible sequences:: + + from librt.random import Random + + def generate_data() -> list[i64]: + rng = Random(42) + return [rng.randint(0, 100) for _ in range(10)] + +Backward compatibility +---------------------- + +New versions of this module are not guaranteed to generate the same results when +using the same seed. A specific seed only produces predictable random numbers on a +specific version of ``librt``. In the future we might provide stronger guarantees. diff --git a/mypyc/ir/deps.py b/mypyc/ir/deps.py index 20b1f102ee383..751845d3a324c 100644 --- a/mypyc/ir/deps.py +++ b/mypyc/ir/deps.py @@ -109,6 +109,7 @@ def get_header(self) -> str: LIBRT_BASE64: Final = Capsule("librt.base64") LIBRT_VECS: Final = Capsule("librt.vecs") LIBRT_TIME: Final = Capsule("librt.time") +LIBRT_RANDOM: Final = Capsule("librt.random") BYTES_EXTRA_OPS: Final = SourceDep("bytes_extra_ops.c") BYTES_WRITER_EXTRA_OPS: Final = SourceDep("byteswriter_extra_ops.c") diff --git a/mypyc/ir/rtypes.py b/mypyc/ir/rtypes.py index 60e9b49582bc3..db29f9e304d8d 100644 --- a/mypyc/ir/rtypes.py +++ b/mypyc/ir/rtypes.py @@ -41,7 +41,7 @@ class to enable the new behavior. In rare cases, adding a new from typing import TYPE_CHECKING, ClassVar, Final, Generic, TypeGuard, TypeVar, Union, final from mypyc.common import HAVE_IMMORTAL, IS_32_BIT_PLATFORM, PLATFORM_SIZE, JsonDict, short_name -from mypyc.ir.deps import LIBRT_STRINGS, LIBRT_VECS, Dependency +from mypyc.ir.deps import LIBRT_RANDOM, LIBRT_STRINGS, LIBRT_VECS, Dependency from mypyc.namegen import NameGenerator if TYPE_CHECKING: @@ -544,10 +544,15 @@ def __hash__(self) -> int: ("librt.strings.BytesWriter", (LIBRT_STRINGS,)), ("librt.strings.StringWriter", (LIBRT_STRINGS,)), ] +} | { + "librt.random.Random": RPrimitive( + "librt.random.Random", is_unboxed=False, is_refcounted=True, dependencies=(LIBRT_RANDOM,) + ) } bytes_writer_rprimitive: Final = KNOWN_NATIVE_TYPES["librt.strings.BytesWriter"] string_writer_rprimitive: Final = KNOWN_NATIVE_TYPES["librt.strings.StringWriter"] +random_rprimitive: Final = KNOWN_NATIVE_TYPES["librt.random.Random"] def is_native_rprimitive(rtype: RType) -> bool: diff --git a/mypyc/irbuild/prepare.py b/mypyc/irbuild/prepare.py index 09bfc8339b404..f143ce1b44025 100644 --- a/mypyc/irbuild/prepare.py +++ b/mypyc/irbuild/prepare.py @@ -182,7 +182,12 @@ def load_type_map(mapper: Mapper, modules: list[MypyFile], deser_ctx: DeserMaps) continue mapper.type_to_ir[node.node] = ir mapper.symbol_fullnames.add(node.node.fullname) - mapper.func_to_decl[node.node] = ir.ctor + # Trait/builtin-base classes have an ir.ctor FuncDecl + # but no emitted CPyDef_, so a cross-group direct + # call would hit an undefined symbol. Mirror the skip + # in prepare_init_method. + if not ir.is_trait and not ir.builtin_base: + mapper.func_to_decl[node.node] = ir.ctor for module in modules: for func in get_module_func_defs(module): diff --git a/mypyc/lib-rt/misc_ops.c b/mypyc/lib-rt/misc_ops.c index 2aaadb2ac47d2..392dba0deca4c 100644 --- a/mypyc/lib-rt/misc_ops.c +++ b/mypyc/lib-rt/misc_ops.c @@ -1281,12 +1281,17 @@ static int CPyImport_SetModuleFile(PyObject *modobj, PyObject *module_name, Py_DECREF(file); return 0; } - // Derive __file__ from the shared library's __file__ (for its - // directory), the module name (dots -> path separators), and the - // extension suffix. E.g. for module "a.b.c", shared lib - // "/path/to/group__mypyc.cpython-312-x86_64-linux-gnu.so", - // suffix ".cpython-312-x86_64-linux-gnu.so": - // => "/path/to/a/b/c.cpython-312-x86_64-linux-gnu.so" + // Derive __file__ from the shared lib's directory, the module + // name, and the extension suffix. Two layouts: + // + // Monolithic: one shared lib above the package tree holds many + // modules, so append the full dotted module path. + // separate=True: each module has its own "__mypyc.so" + // next to the module, so dirname(shared_lib) is already inside + // the parent package. Append only the last segment. + // + // Detect the separate=True case by matching the shared lib's + // basename against "__mypyc". PyObject *derived_file = NULL; if (shared_lib_file != NULL && shared_lib_file != Py_None && PyUnicode_Check(shared_lib_file)) { @@ -1314,30 +1319,65 @@ static int CPyImport_SetModuleFile(PyObject *modobj, PyObject *module_name, if (module_path == NULL) { return -1; } + + // Compute the module's last dotted segment for the separate=True check. + Py_ssize_t name_len = PyUnicode_GetLength(module_name); + Py_ssize_t last_dot = PyUnicode_FindChar(module_name, '.', 0, name_len, -1); + PyObject *last_segment; + if (last_dot >= 0) { + last_segment = PyUnicode_Substring(module_name, last_dot + 1, name_len); + } else { + last_segment = module_name; + Py_INCREF(last_segment); + } + if (last_segment == NULL) { + Py_DECREF(module_path); + return -1; + } + // Compare shared_lib_file basename against "__mypyc". + PyObject *expected_basename = PyUnicode_FromFormat( + "%U__mypyc%U", last_segment, ext_suffix); + PyObject *actual_basename; + if (sep >= 0) { + actual_basename = PyUnicode_Substring(shared_lib_file, sep + 1, sf_len); + } else { + actual_basename = shared_lib_file; + Py_INCREF(actual_basename); + } + int is_per_module_lib = 0; + if (expected_basename != NULL && actual_basename != NULL) { + is_per_module_lib = + (PyUnicode_Compare(expected_basename, actual_basename) == 0); + } + Py_XDECREF(expected_basename); + Py_XDECREF(actual_basename); + // For packages, __file__ should point to __init__, // e.g. "a/b/__init__.cpython-312-x86_64-linux-gnu.so". + PyObject *file_path = is_per_module_lib ? last_segment : module_path; if (sep >= 0) { PyObject *dir = PyUnicode_Substring(shared_lib_file, 0, sep); if (dir != NULL) { if (is_package) { derived_file = PyUnicode_FromFormat( "%U%c%U%c__init__%U", dir, (int)sep_char, - module_path, (int)sep_char, ext_suffix); + file_path, (int)sep_char, ext_suffix); } else { derived_file = PyUnicode_FromFormat( "%U%c%U%U", dir, (int)sep_char, - module_path, ext_suffix); + file_path, ext_suffix); } Py_DECREF(dir); } } else { if (is_package) { derived_file = PyUnicode_FromFormat( - "%U%c__init__%U", module_path, (int)SEP[0], ext_suffix); + "%U%c__init__%U", file_path, (int)SEP[0], ext_suffix); } else { - derived_file = PyUnicode_FromFormat("%U%U", module_path, ext_suffix); + derived_file = PyUnicode_FromFormat("%U%U", file_path, ext_suffix); } } + Py_DECREF(last_segment); Py_DECREF(module_path); } if (derived_file == NULL && !PyErr_Occurred()) { diff --git a/mypyc/lib-rt/random/librt_random.c b/mypyc/lib-rt/random/librt_random.c new file mode 100644 index 0000000000000..7dc590eaa5946 --- /dev/null +++ b/mypyc/lib-rt/random/librt_random.c @@ -0,0 +1,762 @@ +#include "pythoncapi_compat.h" + +#define PY_SSIZE_T_CLEAN +#include +#include +#include + +#ifdef _WIN32 +#include +#else +#include +#endif + +#include "mypyc_util.h" +#include "CPy.h" +#include "librt_random.h" + +// +// ChaCha8 PRNG with forward secrecy +// + +#define CHACHA8_RESEED_INTERVAL 16 + +typedef struct { + uint32_t seed[8]; // 256-bit key + uint32_t buf[16]; // output buffer: one ChaCha8 block + uint32_t counter; // block counter + uint8_t used; // index into buf + uint8_t n; // usable values in buf (8 or 16) + uint8_t blocks_left; // blocks until next reseed +} chacha8_rng; + +static inline uint32_t +rotl32(uint32_t x, int n) { + return (x << n) | (x >> (32 - n)); +} + +#define QUARTERROUND(a, b, c, d) \ + do { \ + a += b; d ^= a; d = rotl32(d, 16); \ + c += d; b ^= c; b = rotl32(b, 12); \ + a += b; d ^= a; d = rotl32(d, 8); \ + c += d; b ^= c; b = rotl32(b, 7); \ + } while (0) + +static void +chacha8_block(const uint32_t seed[8], uint32_t counter, uint32_t out[16]) +{ + // "expand 32-byte k" + uint32_t s[16] = { + 0x61707865, 0x3320646e, 0x79622d32, 0x6b206574, + seed[0], seed[1], seed[2], seed[3], + seed[4], seed[5], seed[6], seed[7], + counter, 0, 0, 0 // counter (low 32), counter (high 32), nonce + }; + + memcpy(out, s, sizeof(uint32_t) * 16); + + // 4 double-rounds = 8 rounds + for (int i = 0; i < 4; i++) { + // Column rounds + QUARTERROUND(out[0], out[4], out[ 8], out[12]); + QUARTERROUND(out[1], out[5], out[ 9], out[13]); + QUARTERROUND(out[2], out[6], out[10], out[14]); + QUARTERROUND(out[3], out[7], out[11], out[15]); + // Diagonal rounds + QUARTERROUND(out[0], out[5], out[10], out[15]); + QUARTERROUND(out[1], out[6], out[11], out[12]); + QUARTERROUND(out[2], out[7], out[ 8], out[13]); + QUARTERROUND(out[3], out[4], out[ 9], out[14]); + } + + // Add original state back (standard ChaCha finalization) + for (int i = 0; i < 16; i++) + out[i] += s[i]; +} + +// Fill entropy from OS via os.urandom(), which handles short reads, +// EINTR, and platform differences internally. +// Returns 0 on success, -1 on failure (with Python exception set). +static int +fill_os_entropy(void *buf, size_t len) +{ + PyObject *os_mod = PyImport_ImportModule("os"); + if (os_mod == NULL) + return -1; + PyObject *bytes = PyObject_CallMethod(os_mod, "urandom", "n", (Py_ssize_t)len); + Py_DECREF(os_mod); + if (bytes == NULL) + return -1; + memcpy(buf, PyBytes_AS_STRING(bytes), len); + Py_DECREF(bytes); + return 0; +} + +static void +chacha8_refill(chacha8_rng *rng) +{ + chacha8_block(rng->seed, rng->counter, rng->buf); + rng->counter++; + rng->used = 0; + rng->blocks_left--; + + if (unlikely(rng->blocks_left == 0)) { + // Forward secrecy reseed: steal last 8 words as new key + memcpy(rng->seed, rng->buf + 8, sizeof(uint32_t) * 8); + rng->n = 8; // only 8 words usable this block + rng->counter = 0; + rng->blocks_left = CHACHA8_RESEED_INTERVAL; + } else { + rng->n = 16; + } +} + +static inline uint32_t +chacha8_next(chacha8_rng *rng) +{ + if (unlikely(rng->used >= rng->n)) + chacha8_refill(rng); + return rng->buf[rng->used++]; +} + +// Return 64 bits of randomness (two consecutive 32-bit words, single bounds check). +static inline uint64_t +chacha8_next64(chacha8_rng *rng) +{ + // Need 2 words available; if fewer than 2, refill first. + if (unlikely(rng->used + 1 >= rng->n)) + // Use two separate calls to handle block boundary correctly. + return ((uint64_t)chacha8_next(rng) << 32) | chacha8_next(rng); + uint32_t hi = rng->buf[rng->used++]; + uint32_t lo = rng->buf[rng->used++]; + return ((uint64_t)hi << 32) | lo; +} + +// Return a uniformly distributed random value in [0, range). +// Use Lemire's nearly divisionless method for small ranges, and a portable +// rejection sampler for larger ranges to avoid non-standard 128-bit arithmetic. +static inline uint64_t +chacha8_next_ranged(chacha8_rng *rng, uint64_t range) +{ + assert(range != 0); + if (likely(range <= UINT32_MAX)) { + // 32-bit Lemire: multiply r * range to get 64-bit product, + // upper 32 bits are the result in [0, range). + uint64_t m = (uint64_t)chacha8_next(rng) * range; + uint32_t lo = (uint32_t)m; + if (unlikely(lo < range)) { + uint32_t thresh = (uint32_t)(-(uint32_t)range) % (uint32_t)range; + while (lo < thresh) { + m = (uint64_t)chacha8_next(rng) * range; + lo = (uint32_t)m; + } + } + return m >> 32; + } + // If range is a power of two, masking produces an unbiased result. + if ((range & (range - 1)) == 0) { + return chacha8_next64(rng) & (range - 1); + } + uint64_t r; + // In unsigned arithmetic, -range is 2**64 - range, so this computes + // 2**64 % range. Rejecting values below this threshold leaves exactly + // floor(2**64 / range) full buckets of size range, avoiding modulo bias. + uint64_t thresh = -range % range; + do { + r = chacha8_next64(rng); + } while (unlikely(r < thresh)); + return r % range; +} + +// Return a random i64 starting at 'start', with 'range' possible values. +// A zero range represents the full 2**64 i64 domain. +static inline int64_t +random_i64_from_range(chacha8_rng *rng, int64_t start, uint64_t range) +{ + uint64_t offset = range == 0 ? chacha8_next64(rng) : chacha8_next_ranged(rng, range); + return (int64_t)((uint64_t)start + offset); +} + +static void +chacha8_reset(chacha8_rng *rng) +{ + rng->counter = 0; + rng->used = 16; // force immediate refill on first call + rng->n = 16; + rng->blocks_left = CHACHA8_RESEED_INTERVAL; +} + +static int +chacha8_init(chacha8_rng *rng) +{ + if (fill_os_entropy(rng->seed, sizeof(rng->seed)) < 0) + return -1; + chacha8_reset(rng); + return 0; +} + +// Seed from an integer by hashing it through ChaCha8 to fill the 256-bit key. +static void +chacha8_seed_int(chacha8_rng *rng, int64_t seed_val) +{ + // Use the integer to construct a simple initial key, then run one + // ChaCha8 block to diffuse it across all 256 bits. + memset(rng->seed, 0, sizeof(rng->seed)); + rng->seed[0] = (uint32_t)(seed_val & 0xFFFFFFFF); + rng->seed[1] = (uint32_t)((uint64_t)seed_val >> 32); + + uint32_t out[16]; + chacha8_block(rng->seed, 0, out); + memcpy(rng->seed, out, sizeof(rng->seed)); + chacha8_reset(rng); +} + +// +// Thread-local global RNG for module-level random()/randint() +// +// thread_local pointer for fast access (direct %fs/%gs-relative load), +// platform TLS key with destructor for cleanup on thread exit. +// + +#ifdef _WIN32 +static __declspec(thread) chacha8_rng *tls_rng = NULL; +#else +static __thread chacha8_rng *tls_rng = NULL; +#endif + +#ifdef _WIN32 +static DWORD tls_key = FLS_OUT_OF_INDEXES; + +static void NTAPI +tls_rng_destructor(void *ptr) +{ + if (ptr != NULL) { + memset(ptr, 0, sizeof(chacha8_rng)); + PyMem_RawFree(ptr); + } +} +#else +static pthread_key_t tls_key; + +static void +tls_rng_destructor(void *ptr) +{ + if (ptr != NULL) { + memset(ptr, 0, sizeof(chacha8_rng)); + PyMem_RawFree(ptr); + } +} +#endif + +static int tls_key_created = 0; + +static int +ensure_tls_key(void) +{ + if (likely(tls_key_created)) + return 0; +#ifdef _WIN32 + tls_key = FlsAlloc(tls_rng_destructor); + if (tls_key == FLS_OUT_OF_INDEXES) { + PyErr_SetString(PyExc_OSError, "FlsAlloc failed"); + return -1; + } +#else + if (pthread_key_create(&tls_key, tls_rng_destructor) != 0) { + PyErr_SetString(PyExc_OSError, "pthread_key_create failed"); + return -1; + } +#endif + tls_key_created = 1; + return 0; +} + +// Get the thread-local RNG, initializing on first use. +// Returns NULL with Python exception set on failure. +static inline chacha8_rng * +get_thread_rng(void) +{ + chacha8_rng *rng = tls_rng; + if (likely(rng != NULL)) + return rng; + + // First use on this thread — allocate and seed + rng = PyMem_RawMalloc(sizeof(chacha8_rng)); + if (rng == NULL) { + PyErr_NoMemory(); + return NULL; + } + if (chacha8_init(rng) < 0) { + PyMem_RawFree(rng); + return NULL; + } + + // Register with platform TLS for destructor +#ifdef _WIN32 + FlsSetValue(tls_key, rng); +#else + pthread_setspecific(tls_key, rng); +#endif + + tls_rng = rng; + return rng; +} + +// Return a random double in [0.0, 1.0) with 53 bits of mantissa precision. +static inline double +random_double_impl(chacha8_rng *rng) +{ + uint64_t r = chacha8_next64(rng); + return (double)(r >> 11) * (1.0 / 9007199254740992.0); // 1/2^53 +} + +// +// Module-level random() and randint() +// + +static PyObject* +module_random(PyObject *module, PyObject *Py_UNUSED(ignored)) +{ + chacha8_rng *rng = get_thread_rng(); + if (rng == NULL) + return NULL; + return PyFloat_FromDouble(random_double_impl(rng)); +} + +// Generate random integer in [a, b] using the given RNG. +static inline PyObject* +randint_impl(chacha8_rng *rng, int64_t a, int64_t b) +{ + uint64_t range = (uint64_t)b - (uint64_t)a + 1; + return PyLong_FromLongLong(random_i64_from_range(rng, a, range)); +} + +static PyObject* +module_randint(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + if (nargs != 2) { + PyErr_Format(PyExc_TypeError, + "randint() takes exactly 2 arguments (%zd given)", nargs); + return NULL; + } + + int64_t a = CPyLong_AsInt64(args[0]); + if (unlikely(a == CPY_LL_INT_ERROR && PyErr_Occurred())) + return NULL; + + int64_t b = CPyLong_AsInt64(args[1]); + if (unlikely(b == CPY_LL_INT_ERROR && PyErr_Occurred())) + return NULL; + + if (a > b) { + PyErr_SetString(PyExc_ValueError, + "empty range for randint()"); + return NULL; + } + + chacha8_rng *rng = get_thread_rng(); + if (rng == NULL) + return NULL; + + return randint_impl(rng, a, b); +} + +// Parse 1 or 2 int args for randrange([start,] stop). +// Sets *a to start (default 0), *b to stop-1. +// Returns 0 on success, -1 on error (with exception set). +static int +parse_randrange_args(PyObject *const *args, Py_ssize_t nargs, + int64_t *a, int64_t *b) +{ + if (nargs == 1) { + *a = 0; + int64_t stop = CPyLong_AsInt64(args[0]); + if (unlikely(stop == CPY_LL_INT_ERROR && PyErr_Occurred())) + return -1; + if (stop <= 0) { + PyErr_SetString(PyExc_ValueError, "empty range for randrange()"); + return -1; + } + *b = stop - 1; + } else if (nargs == 2) { + *a = CPyLong_AsInt64(args[0]); + if (unlikely(*a == CPY_LL_INT_ERROR && PyErr_Occurred())) + return -1; + int64_t stop = CPyLong_AsInt64(args[1]); + if (unlikely(stop == CPY_LL_INT_ERROR && PyErr_Occurred())) + return -1; + if (*a >= stop) { + PyErr_SetString(PyExc_ValueError, "empty range for randrange()"); + return -1; + } + *b = stop - 1; + } else { + PyErr_Format(PyExc_TypeError, + "randrange() takes 1 or 2 arguments (%zd given)", nargs); + return -1; + } + return 0; +} + +static PyObject* +module_randrange(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + int64_t a, b; + if (parse_randrange_args(args, nargs, &a, &b) < 0) + return NULL; + + chacha8_rng *rng = get_thread_rng(); + if (rng == NULL) + return NULL; + + return randint_impl(rng, a, b); +} + +static PyObject* +module_seed(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + if (nargs != 1) { + PyErr_Format(PyExc_TypeError, + "seed() takes exactly 1 argument (%zd given)", nargs); + return NULL; + } + int64_t seed_val = CPyLong_AsInt64(args[0]); + if (unlikely(seed_val == CPY_LL_INT_ERROR && PyErr_Occurred())) + return NULL; + + chacha8_rng *rng = get_thread_rng(); + if (rng == NULL) + return NULL; + + chacha8_seed_int(rng, seed_val); + Py_RETURN_NONE; +} + +// +// Random Python type +// + +typedef struct { + PyObject_HEAD + chacha8_rng rng; +} RandomObject; + +static PyTypeObject RandomType; + +static PyObject* +Random_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +{ + if (type != &RandomType) { + PyErr_SetString(PyExc_TypeError, "Random cannot be subclassed"); + return NULL; + } + + RandomObject *self = (RandomObject *)type->tp_alloc(type, 0); + // Seeding is done in tp_init + return (PyObject *)self; +} + +static int +Random_init(RandomObject *self, PyObject *args, PyObject *kwds) +{ + PyObject *seed_obj = NULL; + + if (!PyArg_ParseTuple(args, "|O", &seed_obj)) { + return -1; + } + + if (kwds != NULL && PyDict_Size(kwds) > 0) { + PyErr_SetString(PyExc_TypeError, + "Random() takes no keyword arguments"); + return -1; + } + + if (seed_obj == NULL || seed_obj == Py_None) { + if (chacha8_init(&self->rng) < 0) + return -1; + } else { + int64_t seed_val = CPyLong_AsInt64(seed_obj); + if (unlikely(seed_val == CPY_LL_INT_ERROR && PyErr_Occurred())) + return -1; + chacha8_seed_int(&self->rng, seed_val); + } + + return 0; +} + +// Internal constructors for capsule API (bypass tp_new/tp_init) + +static PyObject * +Random_internal(void) { + RandomObject *self = (RandomObject *)RandomType.tp_alloc(&RandomType, 0); + if (self == NULL) + return NULL; + if (chacha8_init(&self->rng) < 0) { + Py_DECREF(self); + return NULL; + } + return (PyObject *)self; +} + +static PyObject * +Random_from_seed_internal(int64_t seed_val) { + RandomObject *self = (RandomObject *)RandomType.tp_alloc(&RandomType, 0); + if (self == NULL) + return NULL; + chacha8_seed_int(&self->rng, seed_val); + return (PyObject *)self; +} + +static PyTypeObject * +Random_type_internal(void) { + return &RandomType; +} + +static int64_t +Random_randrange1_internal(PyObject *self, int64_t stop) { + if (unlikely(stop <= 0)) { + PyErr_SetString(PyExc_ValueError, "empty range for randrange()"); + return CPY_LL_INT_ERROR; + } + return (int64_t)chacha8_next_ranged(&((RandomObject *)self)->rng, (uint64_t)stop); +} + +static int64_t +Random_randrange2_internal(PyObject *self, int64_t start, int64_t stop) { + if (unlikely(start >= stop)) { + PyErr_SetString(PyExc_ValueError, "empty range for randrange()"); + return CPY_LL_INT_ERROR; + } + uint64_t range = (uint64_t)stop - (uint64_t)start; + return random_i64_from_range(&((RandomObject *)self)->rng, start, range); +} + +static int64_t +Random_randint_internal(PyObject *self, int64_t a, int64_t b) { + if (unlikely(a > b)) { + PyErr_SetString(PyExc_ValueError, "empty range for randint()"); + return CPY_LL_INT_ERROR; + } + uint64_t range = (uint64_t)b - (uint64_t)a + 1; + return random_i64_from_range(&((RandomObject *)self)->rng, a, range); +} + +static double +Random_random_internal(PyObject *self) { + return random_double_impl(&((RandomObject *)self)->rng); +} + +static PyObject* +Random_randint(RandomObject *self, PyObject *const *args, Py_ssize_t nargs) { + if (nargs != 2) { + PyErr_Format(PyExc_TypeError, + "randint() takes exactly 2 arguments (%zd given)", nargs); + return NULL; + } + + int64_t a = CPyLong_AsInt64(args[0]); + if (unlikely(a == CPY_LL_INT_ERROR && PyErr_Occurred())) + return NULL; + + int64_t b = CPyLong_AsInt64(args[1]); + if (unlikely(b == CPY_LL_INT_ERROR && PyErr_Occurred())) + return NULL; + + if (a > b) { + PyErr_SetString(PyExc_ValueError, + "empty range for randint()"); + return NULL; + } + + return randint_impl(&self->rng, a, b); +} + +static PyObject* +Random_randrange(RandomObject *self, PyObject *const *args, Py_ssize_t nargs) { + int64_t a, b; + if (parse_randrange_args(args, nargs, &a, &b) < 0) + return NULL; + return randint_impl(&self->rng, a, b); +} + +static PyObject* +Random_random(RandomObject *self, PyObject *Py_UNUSED(ignored)) { + return PyFloat_FromDouble(random_double_impl(&self->rng)); +} + +static PyObject* +Random_seed(RandomObject *self, PyObject *const *args, Py_ssize_t nargs) { + if (nargs != 1) { + PyErr_Format(PyExc_TypeError, + "seed() takes exactly 1 argument (%zd given)", nargs); + return NULL; + } + int64_t seed_val = CPyLong_AsInt64(args[0]); + if (unlikely(seed_val == CPY_LL_INT_ERROR && PyErr_Occurred())) + return NULL; + chacha8_seed_int(&self->rng, seed_val); + Py_RETURN_NONE; +} + +static PyMethodDef Random_methods[] = { + {"randint", (PyCFunction) Random_randint, METH_FASTCALL, + PyDoc_STR("Return random integer in range [a, b], including both end points.") + }, + {"randrange", (PyCFunction) Random_randrange, METH_FASTCALL, + PyDoc_STR("Return random integer in range [start, stop).") + }, + {"random", (PyCFunction) Random_random, METH_NOARGS, + PyDoc_STR("Return random float in [0.0, 1.0).") + }, + {"seed", (PyCFunction) Random_seed, METH_FASTCALL, + PyDoc_STR("Seed the random number generator with an integer.") + }, + {NULL} /* Sentinel */ +}; + +static PyTypeObject RandomType = { + .ob_base = PyVarObject_HEAD_INIT(NULL, 0) + .tp_name = "Random", + .tp_doc = PyDoc_STR("Fast random number generator using ChaCha8"), + .tp_basicsize = sizeof(RandomObject), + .tp_itemsize = 0, + .tp_flags = Py_TPFLAGS_DEFAULT, + .tp_new = Random_new, + .tp_init = (initproc) Random_init, + .tp_methods = Random_methods, +}; + +// Module definition + +static PyMethodDef librt_random_module_methods[] = { + {"random", (PyCFunction) module_random, METH_NOARGS, + PyDoc_STR("Return random float in [0.0, 1.0) using thread-local RNG.") + }, + {"randint", (PyCFunction) module_randint, METH_FASTCALL, + PyDoc_STR("Return random integer in range [a, b] using thread-local RNG.") + }, + {"randrange", (PyCFunction) module_randrange, METH_FASTCALL, + PyDoc_STR("Return random integer in range [start, stop) using thread-local RNG.") + }, + {"seed", (PyCFunction) module_seed, METH_FASTCALL, + PyDoc_STR("Seed the thread-local RNG with an integer.") + }, + {NULL, NULL, 0, NULL} +}; + +// Module-level internal functions for mypyc primitives (use thread-local RNG) + +static double +module_random_internal(void) { + chacha8_rng *rng = get_thread_rng(); + if (rng == NULL) + return CPY_FLOAT_ERROR; + return random_double_impl(rng); +} + +static int64_t +module_randint_internal(int64_t a, int64_t b) { + if (unlikely(a > b)) { + PyErr_SetString(PyExc_ValueError, "empty range for randint()"); + return CPY_LL_INT_ERROR; + } + chacha8_rng *rng = get_thread_rng(); + if (rng == NULL) + return CPY_LL_INT_ERROR; + uint64_t range = (uint64_t)b - (uint64_t)a + 1; + return random_i64_from_range(rng, a, range); +} + +static int64_t +module_randrange1_internal(int64_t stop) { + if (unlikely(stop <= 0)) { + PyErr_SetString(PyExc_ValueError, "empty range for randrange()"); + return CPY_LL_INT_ERROR; + } + chacha8_rng *rng = get_thread_rng(); + if (rng == NULL) + return CPY_LL_INT_ERROR; + return (int64_t)chacha8_next_ranged(rng, (uint64_t)stop); +} + +static int64_t +module_randrange2_internal(int64_t start, int64_t stop) { + if (unlikely(start >= stop)) { + PyErr_SetString(PyExc_ValueError, "empty range for randrange()"); + return CPY_LL_INT_ERROR; + } + chacha8_rng *rng = get_thread_rng(); + if (rng == NULL) + return CPY_LL_INT_ERROR; + uint64_t range = (uint64_t)stop - (uint64_t)start; + return random_i64_from_range(rng, start, range); +} + +static int +random_abi_version(void) { + return LIBRT_RANDOM_ABI_VERSION; +} + +static int +random_api_version(void) { + return LIBRT_RANDOM_API_VERSION; +} + +static int +librt_random_module_exec(PyObject *m) +{ + if (ensure_tls_key() < 0) { + return -1; + } + if (PyType_Ready(&RandomType) < 0) { + return -1; + } + if (PyModule_AddObjectRef(m, "Random", (PyObject *) &RandomType) < 0) { + return -1; + } + // Export mypyc internal C API via capsule + static void *librt_random_api[LIBRT_RANDOM_API_LEN] = { + (void *)random_abi_version, + (void *)random_api_version, + (void *)Random_internal, + (void *)Random_from_seed_internal, + (void *)Random_type_internal, + (void *)Random_random_internal, + (void *)Random_randint_internal, + (void *)Random_randrange1_internal, + (void *)Random_randrange2_internal, + (void *)module_random_internal, + (void *)module_randint_internal, + (void *)module_randrange1_internal, + (void *)module_randrange2_internal, + }; + PyObject *c_api_object = PyCapsule_New((void *)librt_random_api, "librt.random._C_API", NULL); + if (PyModule_Add(m, "_C_API", c_api_object) < 0) { + return -1; + } + return 0; +} + +static PyModuleDef_Slot librt_random_module_slots[] = { + {Py_mod_exec, librt_random_module_exec}, +#ifdef Py_MOD_GIL_NOT_USED + {Py_mod_gil, Py_MOD_GIL_NOT_USED}, +#endif + {0, NULL} +}; + +static PyModuleDef librt_random_module = { + .m_base = PyModuleDef_HEAD_INIT, + .m_name = "random", + .m_doc = "Fast random number generation using ChaCha8", + .m_size = 0, + .m_methods = librt_random_module_methods, + .m_slots = librt_random_module_slots, +}; + +PyMODINIT_FUNC +PyInit_random(void) +{ + return PyModuleDef_Init(&librt_random_module); +} diff --git a/mypyc/lib-rt/random/librt_random.h b/mypyc/lib-rt/random/librt_random.h new file mode 100644 index 0000000000000..2eabfbd021bc9 --- /dev/null +++ b/mypyc/lib-rt/random/librt_random.h @@ -0,0 +1,10 @@ +#ifndef LIBRT_RANDOM_H +#define LIBRT_RANDOM_H + +#include + +#define LIBRT_RANDOM_ABI_VERSION 1 +#define LIBRT_RANDOM_API_VERSION 9 +#define LIBRT_RANDOM_API_LEN 13 + +#endif // LIBRT_RANDOM_H diff --git a/mypyc/lib-rt/random/librt_random_api.c b/mypyc/lib-rt/random/librt_random_api.c new file mode 100644 index 0000000000000..157fa82b82eb3 --- /dev/null +++ b/mypyc/lib-rt/random/librt_random_api.c @@ -0,0 +1,45 @@ +#include + +#include "librt_random_api.h" + +void *LibRTRandom_API[LIBRT_RANDOM_API_LEN] = {0}; + +int +import_librt_random(void) +{ + PyObject *mod = PyImport_ImportModule("librt.random"); + if (mod == NULL) + return -1; + Py_DECREF(mod); // we import just for the side effect of making the below work. + void **capsule = (void **)PyCapsule_Import("librt.random._C_API", 0); + if (capsule == NULL) + return -1; + + // Only after version validation succeeds can we safely copy the full table. + int (*abi_version)(void) = (int (*)(void))capsule[0]; + int (*api_version)(void) = (int (*)(void))capsule[1]; + if (abi_version() != LIBRT_RANDOM_ABI_VERSION) { + char err[128]; + snprintf(err, sizeof(err), "ABI version conflict for librt.random, expected %d, found %d", + LIBRT_RANDOM_ABI_VERSION, + abi_version() + ); + PyErr_SetString(PyExc_ValueError, err); + return -1; + } + if (api_version() < LIBRT_RANDOM_API_VERSION) { + char err[128]; + snprintf(err, sizeof(err), + "API version conflict for librt.random, expected %d or newer, found %d (hint: upgrade librt)", + LIBRT_RANDOM_API_VERSION, + api_version() + ); + PyErr_SetString(PyExc_ValueError, err); + return -1; + } + // Provider API version is >= our expected version, which (by the API + // compatibility contract) means it has at least LIBRT_RANDOM_API_LEN + // entries, so this copy is safe. + memcpy(LibRTRandom_API, capsule, sizeof(LibRTRandom_API)); + return 0; +} diff --git a/mypyc/lib-rt/random/librt_random_api.h b/mypyc/lib-rt/random/librt_random_api.h new file mode 100644 index 0000000000000..2794de0dd7e58 --- /dev/null +++ b/mypyc/lib-rt/random/librt_random_api.h @@ -0,0 +1,32 @@ +#ifndef LIBRT_RANDOM_API_H +#define LIBRT_RANDOM_API_H + +#include +#include +#include +#include "librt_random.h" + +int +import_librt_random(void); + +extern void *LibRTRandom_API[LIBRT_RANDOM_API_LEN]; + +#define LibRTRandom_ABIVersion (*(int (*)(void)) LibRTRandom_API[0]) +#define LibRTRandom_APIVersion (*(int (*)(void)) LibRTRandom_API[1]) +#define LibRTRandom_Random_internal (*(PyObject* (*)(void)) LibRTRandom_API[2]) +#define LibRTRandom_Random_from_seed_internal (*(PyObject* (*)(int64_t)) LibRTRandom_API[3]) +#define LibRTRandom_Random_type_internal (*(PyTypeObject* (*)(void)) LibRTRandom_API[4]) +#define LibRTRandom_Random_random_internal (*(double (*)(PyObject*)) LibRTRandom_API[5]) +#define LibRTRandom_Random_randint_internal (*(int64_t (*)(PyObject*, int64_t, int64_t)) LibRTRandom_API[6]) +#define LibRTRandom_Random_randrange1_internal (*(int64_t (*)(PyObject*, int64_t)) LibRTRandom_API[7]) +#define LibRTRandom_Random_randrange2_internal (*(int64_t (*)(PyObject*, int64_t, int64_t)) LibRTRandom_API[8]) +#define LibRTRandom_module_random_internal (*(double (*)(void)) LibRTRandom_API[9]) +#define LibRTRandom_module_randint_internal (*(int64_t (*)(int64_t, int64_t)) LibRTRandom_API[10]) +#define LibRTRandom_module_randrange1_internal (*(int64_t (*)(int64_t)) LibRTRandom_API[11]) +#define LibRTRandom_module_randrange2_internal (*(int64_t (*)(int64_t, int64_t)) LibRTRandom_API[12]) + +static inline bool CPyRandom_Check(PyObject *obj) { + return Py_TYPE(obj) == LibRTRandom_Random_type_internal(); +} + +#endif // LIBRT_RANDOM_API_H diff --git a/mypyc/lib-rt/setup.py b/mypyc/lib-rt/setup.py index 49b6c10201317..371b322ca18b2 100644 --- a/mypyc/lib-rt/setup.py +++ b/mypyc/lib-rt/setup.py @@ -151,5 +151,18 @@ def run(self) -> None: Extension( "librt.time", ["time/librt_time.c"], include_dirs=["."], extra_compile_args=cflags ), + Extension( + "librt.random", + [ + "random/librt_random.c", + "init.c", + "int_ops.c", + "exc_ops.c", + "pythonsupport.c", + "getargsfast.c", + ], + include_dirs=["."], + extra_compile_args=cflags, + ), ] ) diff --git a/mypyc/primitives/librt_random_ops.py b/mypyc/primitives/librt_random_ops.py new file mode 100644 index 0000000000000..6aaee84ecd0d6 --- /dev/null +++ b/mypyc/primitives/librt_random_ops.py @@ -0,0 +1,104 @@ +from mypyc.ir.deps import LIBRT_RANDOM +from mypyc.ir.ops import ERR_MAGIC, ERR_NEVER +from mypyc.ir.rtypes import float_rprimitive, int64_rprimitive, random_rprimitive +from mypyc.primitives.registry import function_op, method_op + +# Random() -- construct with OS entropy +function_op( + name="librt.random.Random", + arg_types=[], + return_type=random_rprimitive, + c_function_name="LibRTRandom_Random_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) + +# Random(seed) -- construct with integer seed +function_op( + name="librt.random.Random", + arg_types=[int64_rprimitive], + return_type=random_rprimitive, + c_function_name="LibRTRandom_Random_from_seed_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) + +# Random.randint(a, b) -- return random integer in [a, b] +method_op( + name="randint", + arg_types=[random_rprimitive, int64_rprimitive, int64_rprimitive], + return_type=int64_rprimitive, + c_function_name="LibRTRandom_Random_randint_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) + +# Random.randrange(stop) -- return random integer in [0, stop) +method_op( + name="randrange", + arg_types=[random_rprimitive, int64_rprimitive], + return_type=int64_rprimitive, + c_function_name="LibRTRandom_Random_randrange1_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) + +# Random.randrange(start, stop) -- return random integer in [start, stop) +method_op( + name="randrange", + arg_types=[random_rprimitive, int64_rprimitive, int64_rprimitive], + return_type=int64_rprimitive, + c_function_name="LibRTRandom_Random_randrange2_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) + +# Random.random() -- return random float in [0.0, 1.0) +method_op( + name="random", + arg_types=[random_rprimitive], + return_type=float_rprimitive, + c_function_name="LibRTRandom_Random_random_internal", + error_kind=ERR_NEVER, + dependencies=[LIBRT_RANDOM], +) + +# Module-level random() -- return random float using thread-local RNG +function_op( + name="librt.random.random", + arg_types=[], + return_type=float_rprimitive, + c_function_name="LibRTRandom_module_random_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) + +# Module-level randrange(stop) -- return random integer using thread-local RNG +function_op( + name="librt.random.randrange", + arg_types=[int64_rprimitive], + return_type=int64_rprimitive, + c_function_name="LibRTRandom_module_randrange1_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) + +# Module-level randrange(start, stop) -- return random integer using thread-local RNG +function_op( + name="librt.random.randrange", + arg_types=[int64_rprimitive, int64_rprimitive], + return_type=int64_rprimitive, + c_function_name="LibRTRandom_module_randrange2_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) + +# Module-level randint(a, b) -- return random integer using thread-local RNG +function_op( + name="librt.random.randint", + arg_types=[int64_rprimitive, int64_rprimitive], + return_type=int64_rprimitive, + c_function_name="LibRTRandom_module_randint_internal", + error_kind=ERR_MAGIC, + dependencies=[LIBRT_RANDOM], +) diff --git a/mypyc/primitives/registry.py b/mypyc/primitives/registry.py index c04b4ff65a757..e22a044d9bb27 100644 --- a/mypyc/primitives/registry.py +++ b/mypyc/primitives/registry.py @@ -403,6 +403,7 @@ def load_global_op(name: str, type: RType, src: str) -> LoadAddressDescription: import mypyc.primitives.dict_ops import mypyc.primitives.float_ops import mypyc.primitives.int_ops +import mypyc.primitives.librt_random_ops import mypyc.primitives.librt_strings_ops import mypyc.primitives.librt_time_ops import mypyc.primitives.librt_vecs_ops diff --git a/mypyc/test-data/irbuild-librt-random.test b/mypyc/test-data/irbuild-librt-random.test new file mode 100644 index 0000000000000..9215c13c88d6e --- /dev/null +++ b/mypyc/test-data/irbuild-librt-random.test @@ -0,0 +1,119 @@ +[case testLibrtRandomConstructor_64bit] +from librt.random import Random + +def make_random() -> Random: + return Random() +[out] +def make_random(): + r0 :: librt.random.Random +L0: + r0 = LibRTRandom_Random_internal() + return r0 + +[case testLibrtRandomConstructorWithSeed_64bit] +from librt.random import Random +from mypy_extensions import i64 + +def make_random_seeded(n: i64) -> Random: + return Random(n) +[out] +def make_random_seeded(n): + n :: i64 + r0 :: librt.random.Random +L0: + r0 = LibRTRandom_Random_from_seed_internal(n) + return r0 + +[case testLibrtRandomRandrange_64bit] +from librt.random import Random +from mypy_extensions import i64 + +def randrange1(r: Random, stop: i64) -> i64: + return r.randrange(stop) +def randrange2(r: Random, start: i64, stop: i64) -> i64: + return r.randrange(start, stop) +[out] +def randrange1(r, stop): + r :: librt.random.Random + stop, r0 :: i64 +L0: + r0 = LibRTRandom_Random_randrange1_internal(r, stop) + return r0 +def randrange2(r, start, stop): + r :: librt.random.Random + start, stop, r0 :: i64 +L0: + r0 = LibRTRandom_Random_randrange2_internal(r, start, stop) + return r0 + +[case testLibrtRandomRandint_64bit] +from librt.random import Random +from mypy_extensions import i64 + +def randint(r: Random, a: i64, b: i64) -> i64: + return r.randint(a, b) +[out] +def randint(r, a, b): + r :: librt.random.Random + a, b, r0 :: i64 +L0: + r0 = LibRTRandom_Random_randint_internal(r, a, b) + return r0 + +[case testLibrtRandomRandom_64bit] +from librt.random import Random + +def rand(r: Random) -> float: + return r.random() +[out] +def rand(r): + r :: librt.random.Random + r0 :: float +L0: + r0 = LibRTRandom_Random_random_internal(r) + return r0 + +[case testLibrtRandomModuleRandom_64bit] +from librt.random import random + +def module_random() -> float: + return random() +[out] +def module_random(): + r0 :: float +L0: + r0 = LibRTRandom_module_random_internal() + return r0 + +[case testLibrtRandomModuleRandint_64bit] +from librt.random import randint +from mypy_extensions import i64 + +def module_randint(a: i64, b: i64) -> i64: + return randint(a, b) +[out] +def module_randint(a, b): + a, b, r0 :: i64 +L0: + r0 = LibRTRandom_module_randint_internal(a, b) + return r0 + +[case testLibrtRandomModuleRandrange_64bit] +from librt.random import randrange +from mypy_extensions import i64 + +def module_randrange1(stop: i64) -> i64: + return randrange(stop) +def module_randrange2(start: i64, stop: i64) -> i64: + return randrange(start, stop) +[out] +def module_randrange1(stop): + stop, r0 :: i64 +L0: + r0 = LibRTRandom_module_randrange1_internal(stop) + return r0 +def module_randrange2(start, stop): + start, stop, r0 :: i64 +L0: + r0 = LibRTRandom_module_randrange2_internal(start, stop) + return r0 diff --git a/mypyc/test-data/run-librt-random.test b/mypyc/test-data/run-librt-random.test new file mode 100644 index 0000000000000..0b34222678018 --- /dev/null +++ b/mypyc/test-data/run-librt-random.test @@ -0,0 +1,344 @@ +[case testRandom_librt] +from typing import Any + +from librt.random import Random, random, randint, randrange, seed +from mypy_extensions import i64 +from testutil import assertRaises + +# +# Random object basics +# + +def test_random_construct() -> None: + r = Random() + assert isinstance(r, Random) + +def test_randint_basic() -> None: + r = Random() + for i in range(100): + val = r.randint(0, 10) + assert 0 <= val <= 10 + +def test_randint_single_value() -> None: + r = Random() + for i in range(10): + assert r.randint(5, 5) == 5 + +def test_randint_negative_range() -> None: + r = Random() + for i in range(100): + val = r.randint(-10, -1) + assert -10 <= val <= -1 + +def test_randint_mixed_range() -> None: + r = Random() + for i in range(100): + val = r.randint(-5, 5) + assert -5 <= val <= 5 + +def test_randint_large_range() -> None: + r = Random() + for i in range(100): + val = r.randint(0, 1000000) + assert 0 <= val <= 1000000 + +def test_randint_produces_different_values() -> None: + r = Random() + values = set() + for i in range(100): + values.add(r.randint(0, 1000000)) + # With range 0-1000000 and 100 samples, we should get at least 2 distinct values + assert len(values) > 1 + +def test_random_basic() -> None: + r = Random() + for i in range(100): + val = r.random() + assert 0.0 <= val < 1.0 + +def test_random_returns_float() -> None: + r = Random() + val = r.random() + assert isinstance(val, float) + +def test_random_produces_different_values() -> None: + r = Random() + values = set() + for i in range(100): + values.add(r.random()) + assert len(values) > 1 + +def test_randrange_one_arg() -> None: + r = Random() + for i in range(100): + val = r.randrange(10) + assert 0 <= val < 10 + +def test_randrange_two_args() -> None: + r = Random() + for i in range(100): + val = r.randrange(5, 15) + assert 5 <= val < 15 + +def test_randrange_negative() -> None: + r = Random() + for i in range(100): + val = r.randrange(-10, 0) + assert -10 <= val < 0 + +def test_randrange_single_value() -> None: + r = Random() + for i in range(10): + assert r.randrange(7, 8) == 7 + +def test_randrange_produces_different_values() -> None: + r = Random() + values = set() + for i in range(100): + values.add(r.randrange(1000000)) + assert len(values) > 1 + +def test_constructor_seed() -> None: + r1 = Random(42) + r2 = Random(42) + vals1 = [r1.randint(0, 1000000) for _ in range(20)] + vals2 = [r2.randint(0, 1000000) for _ in range(20)] + assert vals1 == vals2 + +def test_constructor_seed_different() -> None: + r1 = Random(42) + r2 = Random(43) + vals1 = [r1.randint(0, 1000000) for _ in range(20)] + vals2 = [r2.randint(0, 1000000) for _ in range(20)] + assert vals1 != vals2 + +def test_constructor_none_seed() -> None: + r = Random(None) + val = r.random() + assert 0.0 <= val < 1.0 + +def test_seed_method() -> None: + r = Random(0) + r.seed(42) + vals1 = [r.randint(0, 1000000) for _ in range(20)] + r.seed(42) + vals2 = [r.randint(0, 1000000) for _ in range(20)] + assert vals1 == vals2 + +def test_seed_method_resets_state() -> None: + r = Random(42) + expected = [r.randint(0, 1000000) for _ in range(20)] + # Consume some values, then reseed + r.seed(42) + actual = [r.randint(0, 1000000) for _ in range(20)] + assert expected == actual + +# +# Module-level functions +# + +def test_module_random_basic() -> None: + for i in range(100): + val = random() + assert 0.0 <= val < 1.0 + +def test_module_random_returns_float() -> None: + assert isinstance(random(), float) + +def test_module_random_produces_different_values() -> None: + values = set() + for i in range(100): + values.add(random()) + assert len(values) > 1 + +def test_module_randint_basic() -> None: + for i in range(100): + val = randint(0, 10) + assert 0 <= val <= 10 + +def test_module_randint_single_value() -> None: + for i in range(10): + assert randint(5, 5) == 5 + +def test_module_randint_produces_different_values() -> None: + values = set() + for i in range(100): + values.add(randint(0, 1000000)) + assert len(values) > 1 + +def test_module_randrange_one_arg() -> None: + for i in range(100): + val = randrange(10) + assert 0 <= val < 10 + +def test_module_randrange_two_args() -> None: + for i in range(100): + val = randrange(5, 15) + assert 5 <= val < 15 + +def test_module_randrange_produces_different_values() -> None: + values = set() + for i in range(100): + values.add(randrange(1000000)) + assert len(values) > 1 + +def test_module_seed_reproducible() -> None: + seed(42) + vals1 = [randint(0, 1000000) for _ in range(20)] + seed(42) + vals2 = [randint(0, 1000000) for _ in range(20)] + assert vals1 == vals2 + +def test_module_seed_different() -> None: + seed(42) + vals1 = [randint(0, 1000000) for _ in range(20)] + seed(43) + vals2 = [randint(0, 1000000) for _ in range(20)] + assert vals1 != vals2 + +# +# Wrapper function calling convention (via Any) +# + +def test_method_random_via_wrapper() -> None: + r: Any = Random(42) + val = r.random() + assert isinstance(val, float) + assert 0.0 <= val < 1.0 + +def test_method_seed_via_wrapper() -> None: + r: Any = Random(0) + r.seed(42) + val = r.random() + assert 0.0 <= val < 1.0 + +def test_module_random_via_wrapper() -> None: + random_any: Any = random + val = random_any() + assert isinstance(val, float) + assert 0.0 <= val < 1.0 + +def test_module_randint_via_wrapper() -> None: + randint_any: Any = randint + val = randint_any(0, 10) + assert 0 <= val <= 10 + +def test_module_seed_via_wrapper() -> None: + seed_any: Any = seed + seed_any(42) + +# +# Wide i64 ranges +# + +def method_randint(r: Random, a: i64, b: i64) -> i64: + return r.randint(a, b) + +def method_randrange(r: Random, a: i64, b: i64) -> i64: + return r.randrange(a, b) + +def module_randint(a: i64, b: i64) -> i64: + return randint(a, b) + +def module_randrange(a: i64, b: i64) -> i64: + return randrange(a, b) + +def test_full_i64_randint_native() -> None: + lo: i64 = -9223372036854775808 + hi: i64 = 9223372036854775807 + r = Random(42) + saw_non_min = False + for i in range(20): + val = method_randint(r, lo, hi) + assert lo <= val <= hi + if val != lo: + saw_non_min = True + assert saw_non_min + +def test_full_i64_randint_module_native() -> None: + lo: i64 = -9223372036854775808 + hi: i64 = 9223372036854775807 + saw_non_min = False + for i in range(20): + val = module_randint(lo, hi) + assert lo <= val <= hi + if val != lo: + saw_non_min = True + assert saw_non_min + +def test_wide_i64_randrange_native() -> None: + lo: i64 = -9223372036854775808 + hi: i64 = 9223372036854775807 + r = Random(43) + for i in range(20): + val = method_randrange(r, lo, hi) + assert lo <= val < hi + val = module_randrange(lo, hi) + assert lo <= val < hi + +def test_full_i64_randint_python_api() -> None: + r: Any = Random(42) + lo = -9223372036854775808 + hi = 9223372036854775807 + saw_non_min = False + for i in range(20): + val = r.randint(lo, hi) + assert lo <= val <= hi + if val != lo: + saw_non_min = True + assert saw_non_min + +def test_wide_i64_randrange_python_api() -> None: + r: Any = Random(43) + randrange_any: Any = randrange + lo = -9223372036854775808 + hi = 9223372036854775807 + for i in range(20): + val = r.randrange(lo, hi) + assert lo <= val < hi + val = randrange_any(lo, hi) + assert lo <= val < hi + +# +# Error handling +# + +def test_randint_empty_range() -> None: + r = Random() + with assertRaises(ValueError, "empty range"): + r.randint(10, 5) + +def test_randint_wrong_arg_count() -> None: + r = Random() + with assertRaises(TypeError): + r.randint(1) # type: ignore[call-arg] + with assertRaises(TypeError): + r.randint(1, 2, 3) # type: ignore[call-arg] + +def test_module_randint_empty_range() -> None: + with assertRaises(ValueError, "empty range"): + randint(10, 5) + +def test_randrange_empty_range() -> None: + r = Random() + with assertRaises(ValueError, "empty range"): + r.randrange(0) + with assertRaises(ValueError, "empty range"): + r.randrange(-5) + with assertRaises(ValueError, "empty range"): + r.randrange(10, 10) + with assertRaises(ValueError, "empty range"): + r.randrange(10, 5) + +def test_randrange_wrong_arg_count() -> None: + r = Random() + with assertRaises(TypeError): + r.randrange() # type: ignore[call-overload] + with assertRaises(TypeError): + r.randrange(1, 2, 3) # type: ignore[call-overload] + +def test_module_randrange_empty_range() -> None: + with assertRaises(ValueError, "empty range"): + randrange(0) + with assertRaises(ValueError, "empty range"): + randrange(10, 5) diff --git a/mypyc/test/test_irbuild.py b/mypyc/test/test_irbuild.py index f1f0ec777c3da..7e3993e267e74 100644 --- a/mypyc/test/test_irbuild.py +++ b/mypyc/test/test_irbuild.py @@ -59,6 +59,7 @@ "irbuild-math.test", "irbuild-weakref.test", "irbuild-librt-strings.test", + "irbuild-librt-random.test", "irbuild-base64.test", "irbuild-time.test", "irbuild-match.test", diff --git a/mypyc/test/test_run.py b/mypyc/test/test_run.py index 8fb861f5c2aae..e7be5fcf8425a 100644 --- a/mypyc/test/test_run.py +++ b/mypyc/test/test_run.py @@ -81,6 +81,7 @@ "run-librt-strings.test", "run-base64.test", "run-librt-time.test", + "run-librt-random.test", "run-match.test", "run-vecs-i64-interp.test", "run-vecs-misc-interp.test", diff --git a/pyproject.toml b/pyproject.toml index 9313335b0d969..23824197c748a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -10,7 +10,7 @@ requires = [ "mypy_extensions>=1.0.0", "pathspec>=1.0.0", "tomli>=1.1.0; python_version<'3.11'", - "librt>=0.10.0; platform_python_implementation != 'PyPy'", + "librt>=0.11.0; platform_python_implementation != 'PyPy'", # the following is from build-requirements.txt "types-psutil", "types-setuptools", @@ -57,7 +57,7 @@ dependencies = [ "mypy_extensions>=1.0.0", "pathspec>=1.0.0", "tomli>=1.1.0; python_version<'3.11'", - "librt>=0.10.0; platform_python_implementation != 'PyPy'", + "librt>=0.11.0; platform_python_implementation != 'PyPy'", "ast-serialize>=0.3.0,<1.0.0", ] dynamic = ["version"] diff --git a/setup.py b/setup.py index d36a6bfa2c2dc..1879f6892ba8f 100644 --- a/setup.py +++ b/setup.py @@ -153,6 +153,7 @@ def run(self) -> None: debug_level = os.getenv("MYPYC_DEBUG_LEVEL", "1") force_multifile = os.getenv("MYPYC_MULTI_FILE", "") == "1" log_trace = bool(int(os.getenv("MYPYC_LOG_TRACE", "0"))) + separate = os.getenv("MYPYC_SEPARATE", "") == "1" ext_modules = mypycify( mypyc_targets + ["--config-file=mypy_bootstrap.ini"], opt_level=opt_level, @@ -161,6 +162,7 @@ def run(self) -> None: # our Appveyor builds run out of memory sometimes. multi_file=sys.platform == "win32" or force_multifile, log_trace=log_trace, + separate=separate, # Mypy itself is allowed to use native_internal extension. depends_on_librt_internal=True, ) diff --git a/test-data/unit/check-fastparse.test b/test-data/unit/check-fastparse.test index 0897760fb23a7..67ba0c926f829 100644 --- a/test-data/unit/check-fastparse.test +++ b/test-data/unit/check-fastparse.test @@ -337,7 +337,7 @@ def call() -> str: pass [builtins fixtures/module.pyi] [case testInvalidEscapeSequenceWarningsSuppressed] -# flags: --python-version 3.15 +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # Test that SyntaxWarnings for invalid escape sequences are suppressed # when parsing potential type expressions containing regex patterns or # similar strings. Callable arguments are always potential type expressions. diff --git a/test-data/unit/check-functions.test b/test-data/unit/check-functions.test index bd2bf26613940..893eefb36f874 100644 --- a/test-data/unit/check-functions.test +++ b/test-data/unit/check-functions.test @@ -3814,9 +3814,9 @@ f(1, b'x', 1) main:3: error: Missing positional argument "y" in call to "f" [case testMissingPositionalArgShiftDetectedFirst] -def f(x: int, y: str, z: bytes) -> None: ... +def f(x: int, y: str, z: bytes, last: float) -> None: ... -f("hello", b'x') +f("hello", b'x', 1.5) [builtins fixtures/primitives.pyi] [out] main:3: error: Missing positional argument "x" in call to "f" @@ -3891,3 +3891,46 @@ f("hello", b'x') main:3: error: Missing positional argument "z" in call to "f" main:3: error: Argument 1 to "f" has incompatible type "str"; expected "int" main:3: error: Argument 2 to "f" has incompatible type "bytes"; expected "str" + +[case testMissingPositionalArgNamesHigherN] +# See https://github.com/python/mypy/issues/21427 +def convert2(first: int, second: str) -> None: ... + +# Possibly omitted arg, but we still issue two errors because there is only one argument +convert2("hello") # E: Missing positional argument "second" in call to "convert2" \ + # E: Argument 1 to "convert2" has incompatible type "str"; expected "int" + +# Other cases +convert2() # E: Missing positional arguments "first", "second" in call to "convert2" + +convert2("hello", 1) # E: Argument 1 to "convert2" has incompatible type "str"; expected "int" \ + # E: Argument 2 to "convert2" has incompatible type "int"; expected "str" + +def convert3(first: int, second: str, third: float) -> None: ... + +# Possibly omitted arg, but we now only issue one error +convert3("hello", 3.15) # E: Missing positional argument "first" in call to "convert3" + +# Other cases +convert3("hello") # E: Missing positional arguments "second", "third" in call to "convert3" \ + # E: Argument 1 to "convert3" has incompatible type "str"; expected "int" + +convert3(3.15, "hello") # E: Missing positional argument "third" in call to "convert3" \ + # E: Argument 1 to "convert3" has incompatible type "float"; expected "int" + +def convert4(first: int, second: str, third: float, fourth: bytes) -> None: ... + +# Possibly omitted arg, but we now only issue one error +convert4("hello", 3.15, b'') # E: Missing positional argument "first" in call to "convert4" + +# Other cases +convert4("hello") # E: Missing positional arguments "second", "third", "fourth" in call to "convert4" \ + # E: Argument 1 to "convert4" has incompatible type "str"; expected "int" + +convert4("hello", 3.15) # E: Missing positional arguments "third", "fourth" in call to "convert4" \ + # E: Argument 1 to "convert4" has incompatible type "str"; expected "int" \ + # E: Argument 2 to "convert4" has incompatible type "float"; expected "str" + +convert4(b'', "hello", 3.15) # E: Missing positional argument "fourth" in call to "convert4" \ + # E: Argument 1 to "convert4" has incompatible type "bytes"; expected "int" +[builtins fixtures/primitives.pyi] diff --git a/test-data/unit/check-generics.test b/test-data/unit/check-generics.test index a3a5b02d54f89..b6a97c70f4950 100644 --- a/test-data/unit/check-generics.test +++ b/test-data/unit/check-generics.test @@ -3542,7 +3542,7 @@ reveal_type(C.foo) # N: Revealed type is "def [T] (self: __main__.B[T`1]) -> T` reveal_type(C[int].foo) # N: Revealed type is "def (self: __main__.B[builtins.int]) -> builtins.int" reveal_type(D.foo) # N: Revealed type is "def (self: __main__.B[builtins.int]) -> builtins.int" -[case testDeterminismFromJoinOrderingInSolver] +[case testDeterminismFromJoinOrderingInSolver1] # Used to fail non-deterministically # https://github.com/python/mypy/issues/19121 from __future__ import annotations @@ -3595,6 +3595,46 @@ def draw_none( takes_int_str_none(c3) [builtins fixtures/tuple.pyi] +[case testDeterminismFromJoinOrderingInSolver2] +# Used to fail non-deterministically +# https://github.com/python/mypy/issues/21445 +from typing import Generic, Iterable, TypeVar, overload + +class A: ... + +@overload +def f0(a: A, b: object, /) -> object: ... +@overload +def f0(a: object, b: int, /) -> object: ... +def f0(a, b, /): ... + +@overload +def f1(a: int, b: object, /) -> object: ... +@overload +def f1(a: object, b: A, /) -> object: ... +def f1(a, b, /): ... + +def g(a, b, /): ... + +T = TypeVar("T") +K = TypeVar("K") +V = TypeVar("V") + +class ziplike(Generic[T]): + def __new__(cls, x: str, y: tuple[V, ...], /) -> ziplike[tuple[str, V]]: + raise + def __iter__(self) -> ziplike[T]: + return self + def __next__(self) -> T: + raise + +class dictlike(Generic[K, V]): + def __init__(self, arg: Iterable[tuple[K, V]]) -> None: pass + +x = dictlike(ziplike("012", (f0, f1, g))) +reveal_type(x) # N: Revealed type is "__main__.dictlike[builtins.str, Overload(def (Any, Any) -> Any, def (Any, Any) -> Any, def (Any, Any) -> Any, def (Any, Any) -> Any)]" +[builtins fixtures/dict.pyi] + [case testPropertyWithGenericSetter] from typing import TypeVar diff --git a/test-data/unit/check-incremental.test b/test-data/unit/check-incremental.test index 6911a350376f3..db15b73419109 100644 --- a/test-data/unit/check-incremental.test +++ b/test-data/unit/check-incremental.test @@ -7995,3 +7995,19 @@ import mod [out2] main:2: error: Cannot find implementation or library stub for module named "mod" main:2: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports + +[case testIncrementalFileConfigCommentsStale] +-- When a dependency changes, the importing module becomes stale and is +-- reprocessed via process_stale_scc. As inline config comments are not cached +-- (by design), moving the order of processing the stale SCC can accidentally +-- break file config comments on subsequent runs. +# mypy: disable-error-code="import-not-found" +import nonexistent +import b +[file b.py] +x = 1 +[file b.py.2] +x = "hello" +[builtins fixtures/module.pyi] +[stale b] +[rechecked b] diff --git a/test-data/unit/check-typeform.test b/test-data/unit/check-typeform.test index 220f31b45b512..6d59cb0081115 100644 --- a/test-data/unit/check-typeform.test +++ b/test-data/unit/check-typeform.test @@ -1,6 +1,7 @@ -- TypeForm Type [case testRecognizesUnparameterizedTypeFormInAnnotation] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm = str reveal_type(typx) # N: Revealed type is "TypeForm[Any]" @@ -8,6 +9,7 @@ reveal_type(typx) # N: Revealed type is "TypeForm[Any]" [typing fixtures/typing-full.pyi] [case testRecognizesParameterizedTypeFormInAnnotation] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm[str] = str reveal_type(typx) # N: Revealed type is "TypeForm[builtins.str]" @@ -18,24 +20,28 @@ reveal_type(typx) # N: Revealed type is "TypeForm[builtins.str]" -- Type Expression Location: Assignment [case testCanAssignTypeExpressionToTypeFormVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm[str] = str [builtins fixtures/primitives.pyi] [typing fixtures/typing-full.pyi] [case testCanAssignTypeExpressionToUnionTypeFormVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm[str | None] = str | None [builtins fixtures/primitives.pyi] [typing fixtures/typing-full.pyi] [case testCannotAssignTypeExpressionToTypeFormVariableWithIncompatibleItemType] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm[str] = int # E: Incompatible types in assignment (expression has type "TypeForm[int]", variable has type "TypeForm[str]") [builtins fixtures/primitives.pyi] [typing fixtures/typing-full.pyi] [case testCanAssignValueExpressionToTypeFormVariableIfValueIsATypeForm1] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx1: TypeForm = str typx2: TypeForm = typx1 # looks like a type expression: name @@ -43,6 +49,7 @@ typx2: TypeForm = typx1 # looks like a type expression: name [typing fixtures/typing-full.pyi] [case testCanAssignValueExpressionToTypeFormVariableIfValueIsATypeForm2] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm def identity_tf(x: TypeForm) -> TypeForm: return x @@ -52,6 +59,7 @@ typx2: TypeForm = identity_tf(typx1) # does not look like a type expression [typing fixtures/typing-full.pyi] [case testCannotAssignValueExpressionToTypeFormVariableIfValueIsNotATypeForm] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm val: int = 42 typx: TypeForm = val # E: Incompatible types in assignment (expression has type "int", variable has type "TypeForm[Any]") @@ -59,6 +67,7 @@ typx: TypeForm = val # E: Incompatible types in assignment (expression has type [typing fixtures/typing-full.pyi] [case testCanAssignNoneTypeExpressionToTypeFormVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm = None reveal_type(typx) # N: Revealed type is "TypeForm[Any]" @@ -66,6 +75,7 @@ reveal_type(typx) # N: Revealed type is "TypeForm[Any]" [typing fixtures/typing-full.pyi] [case testCanAssignTypeExpressionToTypeFormVariableDeclaredEarlier] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Type, TypeForm typ: Type typ = int | None # E: Incompatible types in assignment (expression has type "object", variable has type "type[Any]") @@ -75,6 +85,7 @@ typx = int | None [typing fixtures/typing-full.pyi] [case testCanAssignTypeExpressionWithStringAnnotationToTypeFormVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm[str | None] = 'str | None' [builtins fixtures/primitives.pyi] @@ -84,6 +95,7 @@ typx: TypeForm[str | None] = 'str | None' -- Type Expression Location: Function Parameter [case testCanPassTypeExpressionToTypeFormParameterInFunction] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm def is_type(typx: TypeForm) -> bool: return isinstance(typx, type) @@ -92,6 +104,7 @@ is_type(int | None) [typing fixtures/typing-full.pyi] [case testCannotPassTypeExpressionToTypeParameter] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm def is_type(typ: type) -> bool: return isinstance(typ, type) is_type(int | None) # E: Argument 1 to "is_type" has incompatible type "object"; expected "type" @@ -99,6 +112,7 @@ is_type(int | None) # E: Argument 1 to "is_type" has incompatible type "object" [typing fixtures/typing-full.pyi] [case testCanPassTypeExpressionToTypeFormParameterInMethod] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm class C: def is_type(self, typx: TypeForm) -> bool: @@ -108,6 +122,7 @@ C().is_type(int | None) [typing fixtures/typing-full.pyi] [case testCanPassTypeExpressionToTypeFormParameterInOverload] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import overload, TypeForm @overload def is_type(typx: TypeForm) -> bool: ... @@ -120,6 +135,7 @@ is_type(int | None) [typing fixtures/typing-full.pyi] [case testCanPassTypeExpressionToTypeFormParameterInDecorator] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Callable, TypeForm, TypeVar P = TypeVar('P') R = TypeVar('R') @@ -135,6 +151,7 @@ def sum_ints(x: int | None) -> int: [typing fixtures/typing-full.pyi] [case testCanPassTypeExpressionToTypeFormVarargsParameter] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Callable, ParamSpec, TypeForm, TypeVar P = ParamSpec('P') R = TypeVar('R') @@ -150,6 +167,7 @@ def sum_ints(x: int | None, y: int) -> tuple[int, int]: [typing fixtures/typing-full.pyi] [case testCanPassTypeExpressionWithStringAnnotationToTypeFormParameter] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm def is_type(typx: TypeForm) -> bool: return isinstance(typx, type) @@ -161,6 +179,7 @@ is_type('int | None') -- Type Expression Location: Return Statement [case testCanReturnTypeExpressionInFunctionWithTypeFormReturnType] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm def maybe_int_type() -> TypeForm: return int | None @@ -169,6 +188,7 @@ reveal_type(maybe_int_type()) # N: Revealed type is "TypeForm[Any]" [typing fixtures/typing-full.pyi] [case testCanReturnTypeExpressionWithStringAnnotationInFunctionWithTypeFormReturnType] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm def maybe_int_type() -> TypeForm: return 'int | None' @@ -184,6 +204,7 @@ reveal_type(maybe_int_type()) # N: Revealed type is "TypeForm[Any]" -- have the same rich context as SemanticAnalyzer.try_parse_as_type_expression(). [case testTypeExpressionWithoutStringAnnotationRecognizedInOtherSyntacticLocations] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Dict, List, TypeForm list_of_typx: List[TypeForm] = [int | str] dict_with_typx_keys: Dict[TypeForm, int] = { @@ -195,6 +216,7 @@ dict_with_typx_keys[int | str] + 1 [typing fixtures/typing-full.pyi] [case testTypeExpressionWithStringAnnotationNotRecognizedInOtherSyntacticLocations] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Dict, List, TypeForm list_of_typx: List[TypeForm] = ['int | str'] # E: TypeForm containing a string annotation cannot be recognized here. Surround with TypeForm(...) to recognize. \ # E: List item 0 has incompatible type "str"; expected "TypeForm[Any]" @@ -210,6 +232,7 @@ dict_with_typx_keys['int | str'] += 1 # E: TypeForm containing a string annotat [typing fixtures/typing-full.pyi] [case testValueExpressionWithStringInTypeFormContextEmitsConservativeWarning] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Any, Dict, List, TypeForm types: Dict[str, TypeForm] = {'any': Any} # Ensure warning can be ignored if does not apply. @@ -223,6 +246,7 @@ list_of_typx4: List[TypeForm] = [TypeForm('Any')] [typing fixtures/typing-full.pyi] [case testSelfRecognizedInOtherSyntacticLocations] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import List, Self, TypeForm class C: def foo(self) -> None: @@ -236,6 +260,7 @@ typx4: TypeForm = 'Self' # E: Incompatible types in assignment (expression has [typing fixtures/typing-full.pyi] [case testNameOrDottedNameRecognizedInOtherSyntacticLocations] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm import typing from typing import List, TypeForm list_of_typx: List[TypeForm] = [List | typing.Optional[str]] @@ -244,6 +269,7 @@ typx: TypeForm = List | typing.Optional[str] [typing fixtures/typing-full.pyi] [case testInvalidNameOrDottedNameRecognizedInOtherSyntacticLocations] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import List, TypeForm list_of_typx1: List[TypeForm] = [NoSuchType] # E: Name "NoSuchType" is not defined list_of_typx2: List[TypeForm] = [no_such_module.NoSuchType] # E: Name "no_such_module" is not defined @@ -256,6 +282,7 @@ typx2: TypeForm = no_such_module.NoSuchType # E: Name "no_such_module" is not d -- Type Expression Context: Union[TypeForm, ] [case testAcceptsTypeFormLiteralAssignedToUnionOfTypeFormAndNonStr] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx_or_int1: TypeForm[int | None] | int = int | None # No error; interpret as TypeForm typx_or_int2: TypeForm[int | None] | int = str | None # E: Incompatible types in assignment (expression has type "object", variable has type "TypeForm[int | None] | int") @@ -265,6 +292,7 @@ typx_or_int4: TypeForm[int | None] | int = object() # E: Incompatible types in [typing fixtures/typing-full.pyi] [case testAcceptsTypeFormLiteralAssignedToUnionOfTypeFormAndStr] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx_or_str1: TypeForm[int | None] | str = 'int | None' typx_or_str2: TypeForm[int | None] | str = 'str | None' # No error; interpret as str @@ -274,6 +302,7 @@ typx_or_str4: TypeForm[int | None] | str = object() # E: Incompatible types in [typing fixtures/typing-full.pyi] [case testValueExpressionWithStringInTypeFormUnionContextEmitsConservativeWarning1] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import List, TypeForm list_of_typx1: List[TypeForm[int | None] | str] = ['int | None'] # E: TypeForm containing a string annotation cannot be recognized here. Surround with TypeForm(...) to recognize. list_of_typx2: List[TypeForm[int | None] | str] = ['str | None'] # E: TypeForm containing a string annotation cannot be recognized here. Surround with TypeForm(...) to recognize. @@ -281,6 +310,7 @@ list_of_typx2: List[TypeForm[int | None] | str] = ['str | None'] # E: TypeForm [typing fixtures/typing-full.pyi] [case testValueExpressionWithStringInTypeFormUnionContextEmitsConservativeWarning2] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import List, TypeForm list_of_typx3: List[TypeForm[int | None] | int] = ['int | None'] # E: TypeForm containing a string annotation cannot be recognized here. Surround with TypeForm(...) to recognize. \ # E: List item 0 has incompatible type "str"; expected "TypeForm[int | None] | int" @@ -293,6 +323,7 @@ list_of_typx4: List[TypeForm[str | None] | int] = ['str | None'] # E: TypeForm -- Assignability (is_subtype) [case testTypeFormToTypeFormAssignability] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T1] is assignable to TypeForm[T2] iff T1 is assignable to T2. # - In particular TypeForm[Any] is assignable to TypeForm[Any]. from typing_extensions import TypeForm @@ -316,6 +347,7 @@ typx10: TypeForm[int] = ANY_TF # no error [typing fixtures/typing-full.pyi] [case testTypeToTypeFormAssignability] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - Type[C] is assignable to TypeForm[T] iff C is assignable to T. # - In particular Type[Any] is assignable to TypeForm[Any]. from typing import Type, TypeForm @@ -336,6 +368,7 @@ typx8: TypeForm[int] = ANY_T # no error [typing fixtures/typing-full.pyi] [case testTypeFormToTypeAssignability] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T] is NOT assignable to Type[C]. # - In particular TypeForm[Any] is NOT assignable to Type[Any]. from typing import Type, TypeForm @@ -361,6 +394,7 @@ typ10: Type[object] = ANY_TF # E: Incompatible types in assignment (expression # NOTE: This test doesn't involve TypeForm at all, but is still illustrative # when compared with similarly structured TypeForm-related tests above. [case testTypeToTypeAssignability] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - Type[C1] is assignable to Type[C2] iff C1 is assignable to C2. # - In particular Type[Any] is assignable to Type[Any]. from typing import Type @@ -381,6 +415,7 @@ typ8: Type[object] = ANY_T # no error [typing fixtures/typing-full.pyi] [case testTypeFormToObjectAssignability] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T] is assignable to object and Any. from typing import Any, TypeForm INT_TF: TypeForm[int] = int @@ -400,6 +435,7 @@ any3: Any = ANY_TF -- Join (join_types) [case testTypeFormToTypeFormJoin] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T1] join TypeForm[T2] == TypeForm[T1 join T2] from typing_extensions import TypeForm class AB: @@ -415,6 +451,7 @@ reveal_type([A_TF, B_TF][0]) # N: Revealed type is "TypeForm[__main__.AB]" [typing fixtures/typing-full.pyi] [case testTypeToTypeFormJoin] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T1] join Type[T2] == TypeForm[T1 join T2] from typing import Type, TypeForm class AB: @@ -430,6 +467,7 @@ reveal_type([A_T, B_TF][0]) # N: Revealed type is "TypeForm[__main__.AB]" [typing fixtures/typing-full.pyi] [case testTypeFormToTypeJoin] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T1] join Type[T2] == TypeForm[T1 join T2] from typing import Type, TypeForm class AB: @@ -447,6 +485,7 @@ reveal_type([A_TF, B_T][0]) # N: Revealed type is "TypeForm[__main__.AB]" # NOTE: This test doesn't involve TypeForm at all, but is still illustrative # when compared with similarly structured TypeForm-related tests above. [case testTypeToTypeJoin] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - Type[T1] join Type[T2] == Type[T1 join T2] from typing import Type, TypeForm class AB: @@ -465,6 +504,7 @@ reveal_type([A_T, B_T][0]) # N: Revealed type is "type[__main__.AB]" -- Meet (meet_types) [case testTypeFormToTypeFormMeet] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T1] meet TypeForm[T2] == TypeForm[T1 meet T2] from typing import Callable, TypeForm, TypeVar class AB: @@ -483,6 +523,7 @@ reveal_type(f(g)) # N: Revealed type is "TypeForm[__main__.B]" [typing fixtures/typing-full.pyi] [case testTypeToTypeFormMeet] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T1] meet Type[T2] == Type[T1 meet T2] from typing import Callable, Type, TypeForm, TypeVar class AB: @@ -501,6 +542,7 @@ reveal_type(f(g)) # N: Revealed type is "type[__main__.B]" [typing fixtures/typing-full.pyi] [case testTypeFormToTypeMeet] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - TypeForm[T1] meet Type[T2] == Type[T1 meet T2] from typing import Callable, Type, TypeForm, TypeVar class AB: @@ -521,6 +563,7 @@ reveal_type(f(g)) # N: Revealed type is "type[__main__.B]" # NOTE: This test doesn't involve TypeForm at all, but is still illustrative # when compared with similarly structured TypeForm-related tests above. [case testTypeToTypeMeet] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # - Type[T1] meet Type[T2] == Type[T1 meet T2] from typing import Callable, Type, TypedDict, TypeForm, TypeVar class AB(TypedDict): @@ -540,6 +583,7 @@ reveal_type(f(g)) # N: Revealed type is "type[TypedDict({'b': builtins.str, 'c' -- TypeForm(...) Expression [case testTypeFormExpression] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm tf1 = TypeForm(int | str) reveal_type(tf1) # N: Revealed type is "TypeForm[builtins.int | builtins.str]" @@ -557,6 +601,7 @@ tf6: TypeForm = TypeForm(TypeForm(int) | TypeForm(str)) # E: TypeForm argument -- isinstance [case testTypeFormAndTypeIsinstance] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm[str] = str if isinstance(typx, type): @@ -570,6 +615,7 @@ else: -- Type Variables [case testLinkTypeFormToTypeFormWithTypeVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm, TypeVar T = TypeVar('T') def as_typeform(typx: TypeForm[T]) -> TypeForm[T]: @@ -579,6 +625,7 @@ reveal_type(as_typeform(int | str)) # N: Revealed type is "TypeForm[builtins.in [typing fixtures/typing-full.pyi] [case testLinkTypeFormToTypeWithTypeVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Type, TypeForm, TypeVar T = TypeVar('T') def as_type(typx: TypeForm[T]) -> Type[T] | None: @@ -592,6 +639,7 @@ reveal_type(as_type(int)) # N: Revealed type is "type[builtins.int] | None" [typing fixtures/typing-full.pyi] [case testLinkTypeFormToInstanceWithTypeVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm, TypeVar T = TypeVar('T') def as_instance(typx: TypeForm[T]) -> T | None: @@ -605,6 +653,7 @@ reveal_type(as_instance(int)) # N: Revealed type is "builtins.int | None" [typing fixtures/typing-full.pyi] [case testLinkTypeFormToTypeIsWithTypeVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm, TypeVar from typing_extensions import TypeIs T = TypeVar('T') @@ -619,6 +668,7 @@ else: [typing fixtures/typing-full.pyi] [case testLinkTypeFormToTypeGuardWithTypeVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm, TypeVar from typing_extensions import TypeGuard T = TypeVar('T') @@ -636,13 +686,14 @@ else: -- Type Expressions Assignable To TypeForm Variable [case testEveryKindOfTypeExpressionIsAssignableToATypeFormVariable] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm # NOTE: Importing Callable from collections.abc also works OK from typing import ( - Any, Callable, Dict, List, Literal, NoReturn, + Any, Callable, Dict, List, Literal, LiteralString, NoReturn, Optional, ParamSpec, Self, Type, TypeGuard, TypeVar, Union, ) from typing_extensions import ( - Annotated, Concatenate, LiteralString, Never, TypeAlias, TypeForm, TypeIs, + Annotated, Concatenate, Never, TypeAlias, TypeForm, TypeIs, TypeVarTuple, Unpack, ) # @@ -734,6 +785,7 @@ typx = 'int | str' -- Misc [case testTypeFormHasAllObjectAttributesAndMethods] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm typx: TypeForm[int | str] = int | str print(typx.__class__) # OK @@ -757,6 +809,7 @@ class float: pass [typing fixtures/typing-full.pyi] [case testDottedTypeFormsAreRecognized] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing_extensions import TypeForm import typing class C1: @@ -770,6 +823,7 @@ typx2: TypeForm[typing.Any] = typing.Any # OK -- mypy already refused to recognize TypeVars in value expressions before -- the TypeForm feature was introduced. [case testTypeVarTypeFormsAreOnlyRecognizedInStringAnnotation] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Generic, List, TypeForm, TypeVar E = TypeVar('E') class Box(Generic[E]): @@ -781,6 +835,7 @@ class Box(Generic[E]): [typing fixtures/typing-full.pyi] [case testIncompleteTypeFormsAreNotRecognized] +# flags: --python-version 3.14 --enable-incomplete-feature=TypeForm from typing import Optional, TypeForm typx: TypeForm = Optional # E: Incompatible types in assignment (expression has type "int", variable has type "TypeForm[Any]") [builtins fixtures/primitives.pyi] diff --git a/test-requirements.txt b/test-requirements.txt index 8ac31e0b34666..8f88363268cab 100644 --- a/test-requirements.txt +++ b/test-requirements.txt @@ -24,7 +24,7 @@ identify==2.6.15 # via pre-commit iniconfig==2.1.0 # via pytest -librt==0.10.0 ; platform_python_implementation != "PyPy" +librt==0.11.0 ; platform_python_implementation != "PyPy" # via -r mypy-requirements.txt lxml==6.0.2 ; python_version < "3.15" # via -r test-requirements.in