fix: pytester no longer destroys modules owning process global state - #14850
fix: pytester no longer destroys modules owning process global state#14850RonnyPfannschmidt wants to merge 1 commit into
Conversation
Restoring the sys.modules snapshot orphaned atexit and at-fork handlers, and multiprocessing's resource tracker process, while a re-import installed a second copy of the same state. Preserve asyncio, concurrent, execnet, multiprocessing, random and rlcompleter next to zope and readline, matched per package rather than by name prefix. scripts/check-preserved-modules.py audits the list against the stdlib. Closes pytest-dev#14841 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes pytester’s sys.modules snapshot restoration so it no longer drops modules/packages that install process-global state during import (notably multiprocessing.resource_tracker), preventing duplicated global registrations and noisy interpreter-shutdown failures.
Changes:
- Replaces prefix-based preserve logic with package-boundary matching via a preserved top-level package set in
Pytester. - Expands the preserved set to cover additional stdlib/related packages known to install process-global state.
- Adds an audit script (
scripts/check-preserved-modules.py) plus a dedicated tox env to maintain/validate the preserved list, and adds/extends tests + changelog entry.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
tox.ini |
Adds a tox environment to run the preserved-modules audit script. |
testing/test_pytester.py |
Extends snapshot preservation tests and adds a regression test for multiprocessing.resource_tracker surviving teardown. |
src/_pytest/pytester.py |
Introduces a preserved top-level package set and a package-boundary preservation predicate used by SysModulesSnapshot. |
scripts/check-preserved-modules.py |
New maintenance tool to probe modules in subprocesses and report which ones leave process-global state behind after a simulated snapshot-restore. |
changelog/14841.bugfix.rst |
Documents the bugfix and the new audit tooling. |
Suppressed comments (1)
scripts/check-preserved-modules.py:149
- Same issue as above: the restore/dangling scan assumes every
sys.modulesvalue is a module and callsvars(module). If any entry isNone(or otherwise non-module), the script will crash; skip non-modules here too.
for modname, module in list(sys.modules.items()):
for key, value in list(vars(module).items()):
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
bluetech
left a comment
There was a problem hiding this comment.
The code changes LGTM.
But I'm not sure about adding the script, I don't know if it's worth it to maintain these 500 lines for an obscure thing like this. Maybe just stick it in the issue for reference if we ever need to run it again?
|
|
||
| Such a module was re-imported as a second, independent copy, while the state the first copy installed -- ``atexit`` handlers, :func:`os.register_at_fork` handlers, and in the case of :mod:`multiprocessing` a resource tracker server process -- stayed behind with nothing referring to it. The most visible symptom were ``multiprocessing.resource_tracker`` tracebacks written to stderr at interpreter shutdown. | ||
|
|
||
| ``asyncio``, ``concurrent``, ``execnet``, ``multiprocessing``, ``random`` and ``rlcompleter`` are now preserved next to the ``readline`` and ``zope`` entries pytest already had. The list is audited by the new ``scripts/check-preserved-modules.py``, which probes the standard library for modules that cannot be dropped safely. |
There was a problem hiding this comment.
Would remove the note about the script (not user-facing)
| ``asyncio``, ``concurrent``, ``execnet``, ``multiprocessing``, ``random`` and ``rlcompleter`` are now preserved next to the ``readline`` and ``zope`` entries pytest already had. The list is audited by the new ``scripts/check-preserved-modules.py``, which probes the standard library for modules that cannot be dropped safely. | |
| ``asyncio``, ``concurrent``, ``execnet``, ``multiprocessing``, ``random`` and ``rlcompleter`` are now preserved next to the ``readline`` and ``zope`` entries pytest already had. |
|
|
||
| ``asyncio``, ``concurrent``, ``execnet``, ``multiprocessing``, ``random`` and ``rlcompleter`` are now preserved next to the ``readline`` and ``zope`` entries pytest already had. The list is audited by the new ``scripts/check-preserved-modules.py``, which probes the standard library for modules that cannot be dropped safely. | ||
|
|
||
| Preserving now follows package boundaries instead of matching a plain name prefix, so an unrelated top level module such as ``zopelicious`` is no longer preserved by the ``zope`` entry. |
There was a problem hiding this comment.
Would remove this as well, kinda doubt anyone would be affected by this...
Closes #14841.
Pytesterrestores itssys.modulessnapshot on teardown, which destroyed modules that had installed process global state while being imported. The state outlives the module object, and the next import installs it a second time from a second module instance — formultiprocessingthat means an orphaned resource tracker process whose bookkeeping no longer matches the live one, which fails loudly at interpreter shutdown.asyncio,concurrent,execnet,multiprocessing,randomandrlcompleterare now preserved next tozopeandreadline, each with the evidence in a comment. Matching follows package boundaries instead of a name prefix, sozopeliciousis no longer preserved by thezopeentry — and the check is a single set lookup, ~10x cheaper on a path that runs once per module insys.modulesper test.scripts/check-preserved-modules.pyis where the list comes from. It probes each stdlib module (plus extras such asexecnet) in its own subprocess: recordsatexit/os.register_at_fork/threading._register_atexitregistrations with caller attribution, imports the module, replays the snapshot restore, and reports what was left behind.--pythonaudits another interpreter,--checkexits non-zero on anything uncovered. A full scan takes ~6s; the list above is the union over 3.10, 3.12, 3.13 and 3.14 on Linux.It cannot see state installed by C extension modules, or state installed lazily on first use — the tracker process itself is only found because the module registers a fork handler while being imported.
🤖 Generated with Claude Code