Skip to content

feat: refresh all installed integrations after extension add - #5468

Open
mnriem wants to merge 3 commits into
github:mainfrom
mnriem:mnriem-integration-rescaffolding-assessment
Open

mnriem wants to merge 3 commits into
github:mainfrom
mnriem:mnriem-integration-rescaffolding-assessment

Conversation

@mnriem

@mnriem mnriem commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Description

Regenerate every recorded installed integration after successful specify extension add, so workflows can select a non-default coding agent without a separate integration-selection or scaffolding step.

The refresh uses each integration's saved configuration, regenerates core invocation artifacts, composes enabled extension and preset contributions, and refreshes managed settings and native events. Shared templates and scripts remain aligned with the unchanged project default.

Customized outputs and adapter consent are preserved. Incomplete readiness returns failure with the affected integrations, retained installation/completed refreshes, and recovery guidance. The full-refresh policy is limited to extension addition, including accepted replacements; other lifecycle operations retain their existing behavior.

Testing

Set up this checkout's .venv and editable install using the local development guide. Run commands from the checkout root.

  • Tested locally with .venv/bin/specify --help (Windows PowerShell: .\.venv\Scripts\specify.exe --help)
  • Ran relevant existing tests with .venv/bin/python -m pytest tests -q (Windows PowerShell: .\.venv\Scripts\python.exe -m pytest tests -q); use affected test paths for focused checks and LC_ALL=en_US.UTF-8 on macOS
  • Tested with a sample project (if applicable)

Commands and results on macOS, Python 3.13:

  • .venv/bin/specify --help: passed.
  • LC_ALL=en_US.UTF-8 COLUMNS=1000 .venv/bin/python -m pytest tests -q -n 4 --tb=short: 16653 passed, 208 skipped, 65 warnings.
  • uv tool run ruff@0.15.0 check src tests: passed.
  • git diff --check: passed.

A temporary Claude/Codex project exposes all five assess commands in both integrations after addition, without changing the saved default or options. Controlled workflow subprocesses exercise dynamic dispatch to both integrations without preparation steps; no live coding-agent provider was invoked.

Windows, Linux, other Python versions, and Windows PowerShell 5.1 were not run locally; the platform matrix remains a CI requirement. No PowerShell implementation or generated PowerShell functionality was changed.

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: GitHub Copilot using GPT-6.1 Sol generated the implementation, tests, and documentation and ran automated validation, in autonomous agent execution with maintainer-provided scope decisions. The configured reasoning-effort setting is not exposed to the agent. The changes have not been represented as human-authored or line-by-line human-reviewed.

Regenerate managed integration outputs with saved per-integration settings and effective contributions, while preserving the default, customizations, and adapter consent. Report retained partial readiness failures explicitly.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI adaptive review requested due to automatic review settings November 17, 2026 21:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The refresh path conflates stale-deletion protection with merge authorization, allowing event-bearing extensions to silently overwrite user edits in generated files.

7 open issues
What changed in this PR

This PR adds an extension-add-only refresh across recorded Spec Kit integrations, allowing workflows to select non-default agents without a separate setup step.

Changes:

  • Regenerates shared assets, core commands, enabled contributions, settings, and native events using saved integration options.
  • Adds ownership-aware write checks and partial-failure reporting while retaining the project default and adapter consent.
  • Expands regression coverage and documents the new refresh policy.
File Description
tests/​specify_cli/​integrations/​test_refresh.py Adds multi-integration readiness and preservation coverage.
tests/​specify_cli/​integrations/​test_installed_adapters.py Tests trusted adapter refresh and updates lifecycle fixtures.
tests/​specify_cli/​integrations/​test_command_upgrade.py Preserves historical upgrade scenarios under the new policy.
tests/​specify_cli/​integrations/​test_command_install.py Updates expectations for all-installed registration.
tests/​specify_cli/​extensions/​test_command_add.py Isolates source-focused tests from full refresh.
tests/​specify_cli/​extensions/​test_command_add_path_traversal.py Initializes integration state for security cases.
tests/​specify_cli/​extensions/​test_catalog_versions.py Adds initialized projects to version-install tests.
tests/​specify_cli/​bundles/​test_command_install.py Initializes the integration before extension installation.
src/​specify_cli/​shared_infra.py Adds guarded, strict shared-asset refresh.
src/​specify_cli/​presets/​_manager_commands.py Adds strict preset registration checks.
src/​specify_cli/​integrations/​refresh.py Coordinates refresh, ownership checks, and partial results.
src/​specify_cli/​integrations/​manifest.py Applies write validation when recording generated files.
src/​specify_cli/​integrations/​installer.py Generalizes generated-output recovery ownership updates.
src/​specify_cli/​integrations/​hermes/​__init__.py Routes global skill writes through guarded helpers.
src/​specify_cli/​integrations/​base.py Adds validation to shared integration write helpers.
src/​specify_cli/​integrations/​_helpers.py Reuses the runtime integration-option parser.
src/​specify_cli/​integrations/​_file_changes.py Adds content validation before observed writes.
src/​specify_cli/​integration_runtime.py Adds scoped rendering options and write validation.
src/​specify_cli/​extensions/​command_add.py Defers registration and reports refresh outcomes.
src/​specify_cli/​extensions/​__init__.py Supports deferred lifecycle handling and strict registration.
src/​specify_cli/​agents.py Validates rendered artifacts before writing.
src/​specify_cli/​_init_options.py Supports temporary registration options without persistence.
docs/​integrations/​hermes.md Documents conservative global-skill refresh behavior.
docs/​integrations/​cli.md Explains all-installed refresh and recovery.
design/​integration.md Updates integration lifecycle expectations.
design/​extension.md Defines the add-only post-install refresh boundary.

Review effort: Max (adaptive)

Comment thread src/specify_cli/integrations/refresh.py Resolved

# Unregister commands from all AI agents
if registered_commands:
if registered_commands and not self.defer_registration:
Comment thread src/specify_cli/extensions/__init__.py Resolved
Comment thread src/specify_cli/integrations/refresh.py Resolved
Comment on lines +125 to +127
if path.exists() and (path in self.pending or not self._managed(path, relative)):
self.pending[path] = content
return False
Comment thread src/specify_cli/integrations/refresh.py Resolved
Comment thread src/specify_cli/integrations/refresh.py Resolved
Address scoped readiness review findings for generated events, replacement ownership, legacy skills, required settings, and artifact-free integrations.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI adaptive review requested due to automatic review settings November 17, 2026 22:20
@mnriem

mnriem commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed 0684689 to address the five in-scope findings: separate stale protection from settings-merge authorization, retain replacement registration/hash baselines and reconcile obsolete outputs, verify untouched legacy skill renderings before migration, fail readiness when required settings cannot be merged, and skip invocation registration for artifact-free integrations. Unverifiable obsolete legacy outputs are preserved and reported for manual reconciliation. The two development-symlink requests are intentionally declined inline in accordance with the maintainer's explicit scope decision; no cache namespacing or persistent link-ownership machinery was added.

Validation: LC_ALL=en_US.UTF-8 COLUMNS=1000 .venv/bin/python -m pytest tests -q -n 4 --tb=short passed with 16,673 passed and 208 skipped (16,881 total, 20 more than the previous PR head). Pinned Ruff 0.15.0 and git diff --check passed. Validation was local on macOS/Python 3.13; other platforms and live agent-provider execution were not rerun for this review round.

AI disclosure: Posted for @mnriem by GitHub Copilot, model GPT-6.1 Sol, autonomous mode. The review fixes, automated validation, and this summary were AI-authored/executed; no human line-by-line review is claimed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical ownership-validation gaps in integration refresh can overwrite customized skills and delete untracked command files, risking user data loss.

9 open issues
5 previously missed issues

In code that hasn't changed since last review

Medium severity Reconcile disabled legacy registrations during refresh

src/​specify_cli/​extensions/​__init__.py:2004

This skips disabled extensions whose outputs were generated before contribution manifests existed. Those files are recorded only by registry names, so they are also absent from the hash-based stale-cleanup loop in refresh.py, allowing addition to report readiness while the disabled commands remain invocable. Treat disabled registrations as obsolete too: remove hash-verified files and report unverified legacy files for reconciliation.

Medium severity Scaffold extension configuration before readiness can fail

src/​specify_cli/​extensions/​command_add.py:328

If readiness fails, this exit bypasses manager.scaffold_config() even though the extension remains installed and enabled. Its declared configuration templates are therefore missing after a partial installation. Run scaffolding and report its results before the readiness refresh, preserving the caller-owned step described in design/extension.md (“Installation”).

Medium severity Allow validated removal of empty legacy directories

src/​specify_cli/​integrations/​refresh.py:153

An installed Kimi integration with saved --migrate-legacy options and an empty .kimi/skills directory now fails every extension addition. Its setup calls _remove_empty_legacy_directory(), whose removal observation falls through this branch; _managed() always returns false for directories, so even this safe rmdir() raises a customized-output error. Handle validated empty-directory removal separately from unverified recursive deletion.

Medium severity Reconcile contribution hashes after other artifact-generating operations

src/​specify_cli/​integrations/​refresh.py:180

The contribution hashes become stale after other supported lifecycle writers run. For example, add v1, update the extension to v2, then add a changed v3 with --force: extension update regenerates the invocation files but never updates .contributions/<key>.manifest.json, so this merge retains the v1 baseline and rejects the untouched v2 output as customized. For generic integrations, the subsequent setdefault also ignores the current registrar hash that proves ownership. Synchronize this ledger when other lifecycle operations generate artifacts, or reconcile it with current verified registration hashes before deciding overwrite ownership.

Medium severity Refresh or validate script variants for every secondary integration

src/​specify_cli/​integrations/​refresh.py:300

Only the default's script type is refreshed, but each secondary integration is subsequently regenerated with its own saved script type. With a sh default and a py secondary, for example, a missing .specify/scripts/python/setup_plan.py is neither restored nor validated, yet the secondary plan command points there and addition reports success. Please refresh the required script variants while retaining the default's invocation style, or report incomplete readiness when a secondary integration's script targets are missing.

5 resolved since last review

Review effort: Max (adaptive)

Comment thread src/specify_cli/integrations/refresh.py Resolved
Comment thread src/specify_cli/integrations/refresh.py Resolved
Comment on lines +385 to +387
extensions.register_enabled_extensions_for_agent(
key, force=True, strict=True, development_extension=development_extension,
)
Comment thread src/specify_cli/integrations/refresh.py Resolved
Comment on lines +839 to +841
except (json.JSONDecodeError, UnicodeDecodeError, OSError) as exc:
if strict:
raise ValueError(f"Cannot merge required settings: {dst}; reconcile the existing file before retry") from exc
Comment on lines +412 to +422
if strict:
missing = [
template["name"] for template in manifest.templates
if template.get("type") == "command"
and (pack_id, template["name"]) not in successful_command_replacements
and (pack_id, template["name"]) not in successful_skill_replacements
]
if missing:
raise ValueError(
f"Missing preset invocation artifacts for '{agent_name}': {', '.join(missing)}"
)
Comment thread docs/integrations/cli.md
Comment on lines +558 to +563
Successful `specify extension add` refreshes every integration recorded as installed,
using each integration's saved options and output mode. It regenerates core
commands/skills, enabled extension and preset contributions, managed settings and
native events, and shared templates/scripts aligned with the unchanged default.
Agent directories alone do not make an integration eligible. Regeneration is
local; it does not install agent CLIs, upgrade components, or grant adapter trust.
@mnriem mnriem added the triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review label Nov 17, 2026
Preserve customized intermediate-layer matches, reject missing core sources, guard legacy command retirement, and restore registration authority after failed refreshes. Keep supported no-base preset skips nonfatal and align lifecycle documentation.

Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI adaptive review requested due to automatic review settings November 17, 2026 23:08
@mnriem

mnriem commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed a2b942a addressing all seven comments in review 5947877938. Unverified output matches now remain pending until final composition; strict native-skill refresh avoids rendering the same file a second time in a different format, preserving the legacy migration case. Missing core sources fail before cleanup, legacy Qoder retirement uses guarded removal, and failed refreshes restore prior registration authority so a later extension removal cannot delete a preserved untracked file. Supported no-base preset compositions remain warning-and-skip cases without weakening other strict failures. Updated the Copilot settings documentation and the active/default integration descriptions on the three identified pages.

All seven new findings are within the agreed scope. No additional symlink/cache work was introduced; the two previously declined development-symlink conversations were already resolved.

Validation: LC_ALL=en_US.UTF-8 COLUMNS=1000 .venv/bin/python -m pytest tests -q -n 4 --tb=short passed: 16,688 passed, 208 skipped (16,896 total, 15 more than the previous head). The focused refresh file passed all 60 cases. Pinned Ruff 0.15.0 and git diff --check passed. Validation was local on macOS/Python 3.13; other platforms and live agent-provider execution were not rerun for this round.

AI disclosure: Posted for @mnriem by GitHub Copilot, model GPT-6.1 Sol, autonomous mode. The fixes, automated validation, and this summary were AI-authored/executed; no human line-by-line review is claimed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Refresh can write through symlinked destinations and partially migrate legacy Kilo projects before validation fails, leaving unintended filesystem changes.

7 open issues
5 previously missed issues

In code that hasn't changed since last review

High severity Guard legacy Kilo migration before mutation

src/​specify_cli/​integrations/​refresh.py:359

A legacy Kilo project with preset-overridden commands can be left partially migrated: setup creates .kilo/commands, then stale cleanup rejects a changed legacy command and may already have deleted other legacy core files. Apply the pre-mutation migration guard used by integrations/command_upgrade.py:101–150, or exclude legacy command-root migrations from this refresh. Document extension-add migration behavior in docs/integrations/kilocode.md, as required by design/integration.md, “Integration delivery boundary.”

AI disclosure: GitHub Copilot for the review requester; model and reasoning settings unavailable; autonomous, fully AI-authored review.

High severity Reject symlinked output roots before setup

src/​specify_cli/​integrations/​refresh.py:359

Validate the lexical output directory before calling setup(). Built-in setup paths such as SkillsIntegration.setup() resolve the destination first (base.py:2196), so the new guard never sees an in-project directory symlink. If .claude/skills points to another project-local directory, core regeneration creates files in that target before extension registration rejects the symlink; the failed refresh then retains those unintended writes. Reject symlinked destination roots/ancestors before setup resolves or creates them.

AI disclosure: GitHub Copilot for the review requester; model/reasoning setting not exposed; autonomous, fully AI-authored review.

Medium severity Refresh contribution hashes after lifecycle rendering

src/​specify_cli/​integrations/​refresh.py:328

After extension update or integration upgrade, this contribution manifest can retain old hashes even though the operation successfully regenerated the invocation files. A later extension replacement can then reject those unchanged generated files as customized, making extension add --force fail without user edits. Synchronize contribution ownership after successful lifecycle rendering, and cover both sequences with regression tests.

AI disclosure: GitHub Copilot for the review requester; model and reasoning settings unavailable; autonomous, fully AI-authored review.

Medium severity Reconcile overrides for commands without presets

src/​specify_cli/​integrations/​refresh.py:395

[P2] Reconcile project-local overrides even when no preset declares the command. For example, with .specify/templates/overrides/speckit.plan.md in a Claude/Codex project with no presets, setup regenerates the bundled plan, and this preset pass has no affected command names to reconcile. Addition therefore reports success without materializing the highest-priority override. Resolve the final content for all generated core/extension command names, not just preset-declared names, to honor the priority contract in design/extension.md, “Artifact registration and resolution.”

AI disclosure: GitHub Copilot automated review for the requesting reviewer; model/settings are not exposed; this analysis and comment are AI-generated.

Medium severity Skip artifact checks for generic integrations

src/​specify_cli/​presets/​_manager_commands.py:417

An installed generic integration with an enabled command preset makes every extension add fail this check, because the preset registrar intentionally excludes generic and cannot produce the required artifacts. Preserve that existing exclusion by enforcing this check only when agent_config is present, and add a generic-plus-preset regression case.

AI disclosure: GitHub Copilot for the review requester; model and reasoning settings unavailable; autonomous, fully AI-authored review.

3 resolved since last review

Review effort: Max (adaptive)

Comment on lines +429 to +435
restored = {
"registered_commands": metadata.get("registered_commands", {}),
"registered_skills": metadata.get("registered_skills", []),
}
if "generic_artifact_hashes" in metadata:
restored["generic_artifact_hashes"] = metadata["generic_artifact_hashes"]
registry.update(component_id, restored)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants