convoy: flip visual-diff to a hard merge gate (harden-visual-diff-gate brief 2/2) #140

Merged

varutasu merged 1 commit from convoy/harden-visual-diff-gate-brief-2-flip-gate into main

2026-06-12 23:03:17 -04:00

Author	SHA1	Message	Date
Randall Stillwell	ac91d73a34	convoy: flip visual-diff to a hard merge gate (harden-visual-diff-gate brief 2/2) Removes `continue-on-error: true` from `.github/workflows/visual-diff.yml`'s `Capture screenshots (PR)` step. Visual drift is now a real merge gate on UI-touching PRs. Brief 2/2 of the `harden-visual-diff-gate` convoy. PR #138 shipped the seed workflow (Brief 1); PR #139 (`54495fe`) landed the fresh Linux baseline regenerated against post-glass-redesign main on CT 111. With a known-good baseline committed, the gate can flip without false-failing every UI-touching PR. ## What changes - `.github/workflows/visual-diff.yml` — drop the `continue-on-error: true` flag; add an inline rationale block linking to the convoy + the operator runbook for both intentional changes (dispatch seed workflow → manually open PR → merge → re-run) and unintentional regressions (inspect artifact diff → fix → push). - `.github/workflows/ci.yml` — add 9th `forbidden-patterns` check that greps `visual-diff.yml` for `^\scontinue-on-error:\strue` and fails the build if it returns. Risk #3 of the convoy made concrete: prevents silent re-introduction via template revert. Scoped narrowly to that one file; other workflows (`seed-visual-baselines.yml`'s PR-open step, etc.) legitimately use the flag. Job name bumped from "Forbidden patterns (8 checks)" → "(9 checks)". All `Check N/8` group labels renumbered to `N/9`. - `AGENTS.md` — § Testing § Visual baselines rewritten to drop the "Known staleness as of 2026-06-12" callout (resolved by PR #139); § Testing § Screenshot diff rewritten to lead with "hard merge gate", document the intentional-change runbook, reference the new ci.yml check, and explicitly mention the org-setting caveat for the seed workflow's auto-PR step. - `tests/visual/homepage.spec.ts` — module docblock rewritten to match the AGENTS.md runbook: drops the "advisory, not gating" language; promotes the seed-visual-baselines workflow as the primary re-seeding path; demotes the Playwright Docker image to the offline fallback. - `.github/workflows/seed-visual-baselines.yml` — patches the `peter-evans/create-pull-request@v6` PR-open failure case discovered during Brief 1's first dispatch (run 27454132468). The PR-open step is now `continue-on-error: true` (narrowly scoped, with an inline rationale callout distinguishing it from the just-removed `visual-diff.yml` flag — that one silently hid real UI regressions; this one fronts a known org-level "Allow GitHub Actions to create and approve pull requests" limitation with a loud failure notice). New steps disambiguate the three possible outcomes (no-changes / pr-opened / branch-pushed-pr-blocked) via a `git ls-remote` check on the bot branch and exit non-zero on the blocked-PR case so the workflow run shows red and the operator gets the exact `gh pr create` command in the run logs. - `.convoys/harden-visual-diff-gate.md` — status: shipping; Step 2 marked SHIPPED; Decision D4 ratified (chose option C: accept org setting, document manual `gh pr create` fallback). Inline links to PR #139 + PR #140. ## Test plan - [x] `npm run lint` — clean (1 pre-existing unrelated warning) - [x] `npm run test:run` — 24 files / 118 tests pass - [ ] CI on this PR: 9th forbidden-patterns check passes; visual-diff job passes against the fresh baseline; convoy-metrics-gate passes (2 new rows added by this commit) - [ ] After merge: smoke test the 9th check by opening a throwaway PR that re-adds `continue-on-error: true` to `visual-diff.yml`; confirm it red-X's. (Skip if confident in the grep.) ## Convoy state - Brief 1: SHIPPED (PR #138, `c100c5f`, 2026-06-13) - Baseline refresh: SHIPPED (PR #139, `54495fe`, 2026-06-13) - Brief 2 (this PR): shipping - Convoy closeout: this PR's merge Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-12 21:58:54 -05:00

Author

SHA1

Message

Date

Randall Stillwell

ac91d73a34

convoy: flip visual-diff to a hard merge gate (harden-visual-diff-gate brief 2/2)

Removes `continue-on-error: true` from `.github/workflows/visual-diff.yml`'s
`Capture screenshots (PR)` step. Visual drift is now a real merge gate
on UI-touching PRs.

Brief 2/2 of the `harden-visual-diff-gate` convoy. PR #138 shipped the
seed workflow (Brief 1); PR #139 (`54495fe`) landed the fresh Linux
baseline regenerated against post-glass-redesign main on CT 111. With a
known-good baseline committed, the gate can flip without false-failing
every UI-touching PR.

## What changes

- `.github/workflows/visual-diff.yml` — drop the
  `continue-on-error: true` flag; add an inline rationale block linking
  to the convoy + the operator runbook for both intentional changes
  (dispatch seed workflow → manually open PR → merge → re-run) and
  unintentional regressions (inspect artifact diff → fix → push).
- `.github/workflows/ci.yml` — add 9th `forbidden-patterns` check that
  greps `visual-diff.yml` for `^\s*continue-on-error:\s*true` and fails
  the build if it returns. Risk #3 of the convoy made concrete: prevents
  silent re-introduction via template revert. Scoped narrowly to that
  one file; other workflows (`seed-visual-baselines.yml`'s PR-open
  step, etc.) legitimately use the flag. Job name bumped from
  "Forbidden patterns (8 checks)" → "(9 checks)". All `Check N/8`
  group labels renumbered to `N/9`.
- `AGENTS.md` — § Testing § Visual baselines rewritten to drop the
  "Known staleness as of 2026-06-12" callout (resolved by PR #139);
  § Testing § Screenshot diff rewritten to lead with "hard merge gate",
  document the intentional-change runbook, reference the new ci.yml
  check, and explicitly mention the org-setting caveat for the seed
  workflow's auto-PR step.
- `tests/visual/homepage.spec.ts` — module docblock rewritten to match
  the AGENTS.md runbook: drops the "advisory, not gating" language;
  promotes the seed-visual-baselines workflow as the primary
  re-seeding path; demotes the Playwright Docker image to the offline
  fallback.
- `.github/workflows/seed-visual-baselines.yml` — patches the
  `peter-evans/create-pull-request@v6` PR-open failure case discovered
  during Brief 1's first dispatch (run 27454132468). The PR-open step
  is now `continue-on-error: true` (narrowly scoped, with an inline
  rationale callout distinguishing it from the just-removed
  `visual-diff.yml` flag — that one silently hid real UI regressions;
  this one fronts a known org-level "Allow GitHub Actions to create
  and approve pull requests" limitation with a loud failure notice).
  New steps disambiguate the three possible outcomes (no-changes /
  pr-opened / branch-pushed-pr-blocked) via a `git ls-remote` check on
  the bot branch and exit non-zero on the blocked-PR case so the
  workflow run shows red and the operator gets the exact `gh pr create`
  command in the run logs.
- `.convoys/harden-visual-diff-gate.md` — status: shipping; Step 2
  marked SHIPPED; Decision D4 ratified (chose option C: accept org
  setting, document manual `gh pr create` fallback). Inline links to
  PR #139 + PR #140.

## Test plan

- [x] `npm run lint` — clean (1 pre-existing unrelated warning)
- [x] `npm run test:run` — 24 files / 118 tests pass
- [ ] CI on this PR: 9th forbidden-patterns check passes; visual-diff
  job passes against the fresh baseline; convoy-metrics-gate passes
  (2 new rows added by this commit)
- [ ] After merge: smoke test the 9th check by opening a throwaway PR
  that re-adds `continue-on-error: true` to `visual-diff.yml`; confirm
  it red-X's. (Skip if confident in the grep.)

## Convoy state

- Brief 1: SHIPPED (PR #138, `c100c5f`, 2026-06-13)
- Baseline refresh: SHIPPED (PR #139, `54495fe`, 2026-06-13)
- Brief 2 (this PR): shipping
- Convoy closeout: this PR's merge

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-06-12 21:58:54 -05:00