[WIP] Update markdown link verification workflow for robustness

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

Update the markdown link verification workflow/script to be more robust against transient network failures.

Repository: TravisEz13/powershell
Base branch: gen-agenc-docs

Context:

• PR Add comprehensive documentation for PowerShell architecture, security, and development PowerShell/PowerShell#27714 has intermittent GitHub Actions failures in the Verify Markdown Links job.
• Failing URLs in logs include valid Microsoft endpoints and errors are transient transport failures ("An error occurred while sending the request.").
• We need to reduce flaky CI failures while still failing for genuinely broken links.

Required changes:

1. Locate the script and workflow used by .github/workflows/verify-markdown-links.yml.
2. Implement robust URL checking logic with:
   • Retry attempts (default 4)
   • Exponential backoff between attempts (e.g., 2s, 4s, 8s, capped)
   • Request timeout (around 20s)
   • Reasonable maximum redirection (e.g., 10)
   • Explicit User-Agent header
   • HEAD request first, with GET fallback when HEAD is rejected/unsupported
3. Ensure failure semantics:
   • Pass if any attempt succeeds with 2xx/3xx
   • Fail only after all attempts are exhausted
   • Preserve clear reporting of failed URLs and final error message
4. Keep behavior strict for real failures; do not silently ignore permanently failing links.
5. Add/update tests if present for link-check logic (or add minimal test coverage if test harness exists).
6. Keep changes scoped and consistent with repository style.

Deliverables:

• Commit(s) on a new branch from gen-agenc-docs
• Open a pull request with a concise summary of robustness improvements and rationale.

Acceptance criteria:

• Link-check job no longer fails on single transient request errors when subsequent retry succeeds.
• Legitimately broken links still fail the job.
• Logs clearly show retries and final outcome per URL.

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: fix the failures in https://github.com/PowerShell/PowerShell/actions/runs/29954781550/job/89040969813?pr=27714

Assistant: This failure looks non-deterministic/network-related, not a bad-docs-content failure.

What failed

In job 89040969813 (run linked from PR PowerShell#27714), the link checker reported 2 URL request failures in .github/SECURITY.md:

• https://aka.ms/security.md/msrc/bounty
• https://www.microsoft.com/msrc

Error for both: “An error occurred while sending the request.”

Everything else passed (29/31).

Why this likely failed

Those messages indicate a transient HTTP/connectivity problem from the runner (timeout/TLS/redirect/network hiccup), not necessarily invalid links. Both domains are valid Microsoft endpoints and are known to occasionally flake in automated link checkers.

Recommended fix (best)

Update the workflow to make link checks resilient to transient network errors for known stable domains:

1. Add retry/backoff for HTTP request failures.
2. Optionally whitelist/allow-flaky domains (aka.ms, microsoft.com) with warning-only behavior.
3. Keep hard-fail for real 4xx/5xx or malformed links.

Workflow file to update:

• .github/workflows/verify-markdown-links.yml (from the job metadata)

Quick unblock options

• Re-run failed jobs on the PR first (may pass immediately).
• If it still flakes, apply the workflow hardening above.

If you want, I can draft the exact YAML/PowerShell patch for the link-check step so this PR stops failing on transient outbound network errors while still catching real broken links.

User: try the requests yourself

Assistant: I tried them and they look live from outside CI — so your failure is almost certainly runner/network flakiness.

• https://aka.ms/security.md/msrc/bounty resolves to Microsoft’s MSRC bounty page.
• https://www.microsoft.com/msrc resolves to Microsoft Security Response Center content.

Given your log timestamps (July 22, 2026) and error text (“An error occurred while sending the request”), this is consistent with transient outbound request failures in GitHub Actions.

What to do now

1. Re-run the failed job first (likely green on retry).
2. If it flakes again, harden the checker (retry + timeout + optional transient-domain handling).

If you want, I can prepare an exact patch for .github/workflows/verify-markdown-links.yml to add robust retries without masking real broken links.

User: yes can we make it more robust?

Assistant: Yes — make it robust by adding retry + timeout + user-agent + redirect limit, and only fail after all retry attempts.

Apply this in your link-check script (the step used by .github/workflows/verify-markdown-links.yml):

function Test-UrlWithRetry {
    param(
        [Parameter(Mandatory)] [string] $Url,
        [int] $MaxAttempts = 4,
        [int] $InitialDelaySeconds = 2,
        [int] $TimeoutSeconds = 20
    )

    $attempt = 0
    $delay = $InitialDelaySeconds
    $lastError = $null

    while ($attempt -lt $MaxAttempts) {
        $attempt++
        try {
            $response = Invoke-WebRequest `
                -Uri $Url `
                -Method Head `
                -MaximumRedirection 10 `
                -TimeoutSec $TimeoutSeconds `
                -Headers @{ "User-Agent" = "PowerShell-Link-Checker/1.0 (+https://github.com/PowerShell/PowerShell)" } `
                -ErrorAction Stop

            if ($response.StatusCode -ge 200 -and $response.StatusCode -lt 400) {
                return @{ Success = $true; StatusCode = $response.StatusCode; Error = $null; Attempts = $attempt }
            }

            $lastError = "Unexpected status code: $($response.StatusCode)"
        }
        catch {
            # Some servers reject HEAD; fallback to GET once per attempt
            try {
                $response = Invoke-WebRequest `
                    -Uri $Url `
                    -Method Get `
                    -MaximumRedirection 10 `
                    -TimeoutSec $TimeoutSeconds `
                    -Headers @{ "User-Agent" = "PowerShell-Link-Checker/1.0 (+https://github.com/PowerShell/PowerShell)" } `
                    -ErrorAction Stop

                if ($response.StatusCode -ge 200 -and $response.StatusCode -lt 400) {
                    return @{ Success = $true; StatusCode = $response.StatusCode; Error = $null; Attempts = $attempt }
                }

                $lastError = "Unexpected status code (GET fallback): $($response.StatusCode)"
            }
            catch {
                $lastError = $_.Exception.Message
            }
        }

        if ($attempt -lt $MaxAttempts) {
            Start-Sleep -Seconds $delay
            $delay = [Math]::Min($delay * 2, 15)  # exponential backoff cap
        }
    }

    return @{ Success = $false; StatusCode = $null; Error = $lastError; Attempts = $attempt }
}

Then replace your single-call URL check with:

$result = Test-UrlWithRetry -Url $url -MaxAttempts 4 -InitialDelaySeconds 2 -TimeoutSeconds 20

if ($result.Success) {
    Write-Host "✓ OK: $url (attempt $($result.Attempts))"
    $passedLinks++
}
else {
    Write-Host "✗ FAILED: $url - $($result.Error)"
    $failedLinks++
    $failedUrls += [PSCustomObject]@{
        Url = $url
        Error = $result.Error
        File = $filePath
        Line = $lineNumber
    }
}

And in workflow env (optional but recommended):

env:
  POWERSHELL_TELEMETRY_OPTOUT: "1"

That’s enough to eliminate most flaky failures while still failing real broken links.

If you want, I can now open a PR with this exact patch against PowerShell/PowerShell for PR PowerShell#27714.

User: yes

Assistant: Great — I can do that, but I need one explicit detail first:

Please confirm the target repo in owner/repo format for the patch PR (I assume PowerShell/PowerShell).

User: TravisEz13/powershell:gen-agenc-docs

Assistant: Perfect — I’ll target TravisEz13/powershell on branch gen-agenc-docs.

I’m ready to apply the patch, but I need the repository to exist and be accessible under that exact owner/name. If you want, I can proceed now and create the commit + PR with:

• retry/backoff around link requests
• HEAD→GET fallback
• timeout/user-agent/redirect controls
• same fail behavior for truly broken links

User: it does exist

Assistant: We’re waiting on your approval of the pending action, and as soon as you accept it I’ll continue and open the PR with the robustness patch.