9c723aaae9
PR #58 (83a358b, 2026-06-02) committed the first Linux visual baseline,
resolving the seed-visual-baselines-on-linux convoy. But the cleanup
sweep across docs that referenced the convoy as "queued / not yet done"
never landed. Three files still describe the world as if PR #58 hadn't
happened, which confuses any agent reading the codebase to understand
the visual-diff pipeline:
1. tests/visual/homepage.spec.ts module docblock — described "FIRST
RUN (no committed baseline yet)" and "SEEDING THE BASELINE
(post-merge follow-up)" as the active state.
2. playwright.config.js snapshotPathTemplate comment — said
"Per Decision 4, we don't commit baselines this convoy" and
pointed at the queued seed convoy as future work.
3. AGENTS.md § Testing § Visual baselines + § CI behavior §
Screenshot diff — claimed "none committed yet" and that the first
visual-diff run "will fail at the test step because no baseline
exists yet."
Sweeping all three to describe the current reality. The Mac-vs-Linux
platform footgun (snapshotPathTemplate has no {platform} token) is
still live, so that warning stays — just rephrased from "don't have
baselines yet" to "don't regenerate them on a Mac."
Also surfaces a separate finding the seed work left behind:
visual-diff.yml's screenshot capture step still carries
`continue-on-error: true`, making the diff advisory rather than gating.
Flipping it requires re-seeding the baseline against post-glass-redesign
main first (the PR #58 baseline predates unify-glass-panel-surfaces +
cleanup-card-item-list-and-share-modal-palette +
migrate-button-input-mobilenav-to-glass-primitive). New convoy seed at
.convoys/harden-visual-diff-gate.md captures the two-step shape (re-seed
baseline, then flip the gate) plus the recommended workflow_dispatch
approach for repeatable re-seeding on CT 111.
No code behavior changes. Documentation + .md convoy seed only.
Co-authored-by: Cursor <cursoragent@cursor.com>
112 lines
5.4 KiB
Markdown
112 lines
5.4 KiB
Markdown
---
|
|
slug: harden-visual-diff-gate
|
|
status: queued
|
|
opened: 2026-06-12
|
|
owner: rstillw
|
|
prerequisites:
|
|
- Stable `main` post-`unify-glass-panel-surfaces` /
|
|
`cleanup-card-item-list-and-share-modal-palette` /
|
|
`migrate-button-input-mobilenav-to-glass-primitive` (all merged)
|
|
- CT 111 self-hosted runner online (resolved by PR #132, 2026-06-07)
|
|
related:
|
|
- PR #58 (`83a358b`) — initial Linux baseline seeded
|
|
- PR #18 (`7b6f751`) — `adopt-playwright-smoke` Decision 4: `continue-on-error` end state
|
|
- `tests/visual/homepage.spec.ts` module docblock
|
|
- `.github/workflows/visual-diff.yml` line ~126
|
|
---
|
|
|
|
# harden-visual-diff-gate
|
|
|
|
## Problem
|
|
|
|
`.github/workflows/visual-diff.yml` still carries `continue-on-error: true`
|
|
on the "Capture screenshots (PR)" step. This was the documented end state
|
|
of `adopt-playwright-smoke` (Decision 4) because no Linux baseline existed
|
|
yet. PR #58 seeded the baseline on 2026-06-02, but the gate was never
|
|
flipped — visual drift continues to ship as artifacts + PR comments only,
|
|
not as a blocking check.
|
|
|
|
Additionally, the committed baseline (`tests/visual/__screenshots__/home.png`)
|
|
was generated against `main` at `83a358b`, which is several convoys behind
|
|
the current homepage rendering (glass redesign briefs, corner-light tone
|
|
adjustments, card-class retirement, etc.). Flipping `continue-on-error`
|
|
today would fail every PR touching `pages/**` / `components/**` / `styles/**`
|
|
against this stale reference.
|
|
|
|
## Two-step shape
|
|
|
|
This is one convoy with a strict ordering constraint:
|
|
|
|
### Step 1 — Re-seed the baseline against current `main`
|
|
|
|
Generate a fresh `home.png` from the current homepage rendering on Linux
|
|
(byte-equivalent to CT 111's runner output). Two options:
|
|
|
|
- **Option A: Ad-hoc on CT 111.** SSH the homelab, exec into a runner
|
|
container, run `BASE_URL=https://deckhearth.com VERCEL_AUTOMATION_BYPASS_SECRET=<value> npm run test:visual:update`,
|
|
scp the new PNG back, commit it. Most direct.
|
|
- **Option B (recommended): New `seed-visual-baselines.yml` workflow.**
|
|
`workflow_dispatch`-only, runs on `[self-hosted, axiom]`, takes
|
|
`BASE_URL` as an input, runs `--update-snapshots`, uploads the
|
|
regenerated `tests/visual/__screenshots__/**` as an artifact for the
|
|
operator to download + commit. Repeatable; no SSH dance.
|
|
|
|
Either option ships a PR with the new baseline PNG. The PR should also
|
|
clear the "Known staleness" callout in `AGENTS.md` § Testing § Visual
|
|
baselines.
|
|
|
|
### Step 2 — Flip the gate
|
|
|
|
After Step 1's PR merges and a follow-up PR confirms `Screenshot diff`
|
|
passes cleanly against the new baseline:
|
|
|
|
- Remove `continue-on-error: true` from
|
|
`.github/workflows/visual-diff.yml` (line ~126).
|
|
- Update `AGENTS.md` § CI behavior § Screenshot diff to drop the
|
|
"advisory, not gating" language.
|
|
- Update the module docblock in `tests/visual/homepage.spec.ts` to
|
|
drop the "still carries `continue-on-error: true`" callout.
|
|
- Optional: bump `expect.toHaveScreenshot.maxDiffPixelRatio` in
|
|
`playwright.config.js` if some pixel jitter is unavoidable (e.g.
|
|
font subpixel rendering across runner image bumps). Default is 0
|
|
(any pixel diff fails); 0.01 (1%) is a common safe floor for
|
|
Chromium-vs-Chromium runs.
|
|
|
|
## Decisions to ratify
|
|
|
|
- **D1.** Option B (new `seed-visual-baselines.yml` workflow) vs Option A
|
|
(ad-hoc). Recommend B for repeatability — the homepage will continue
|
|
to evolve and re-seeding will be a recurring operation, not a
|
|
one-shot.
|
|
- **D2.** `maxDiffPixelRatio` value. Defer to operator preference; 0 is
|
|
the strictest and what we currently use implicitly via Playwright
|
|
defaults. 0.001-0.01 is a reasonable cushion.
|
|
- **D3.** Do we also harden `preview-smoke.yml`? It already runs as a
|
|
blocking gate (no `continue-on-error`); no change needed. This convoy
|
|
is scoped to `visual-diff.yml` only.
|
|
|
|
## Risks
|
|
|
|
| # | Risk | Mitigation |
|
|
|---|---|---|
|
|
| 1 | Re-seeded baseline drifts on next runner-image update | Watchtower's weekly update cycle (CT 111) could shift Chromium subpixel rendering. Mitigate via D2's `maxDiffPixelRatio` cushion |
|
|
| 2 | Step 1's baseline PR fails its own `Screenshot diff` | Expected — the new baseline IS the visual change. Use `pipeline: skip visual` directive in the PR body OR open the baseline PR with the `skip-metrics` label equivalent for visual-diff (currently none — would need a new bypass mechanism). Cleaner: land the baseline PR via the `tests/visual/**` path filter, which DOES trigger visual-diff but the new screenshot vs new baseline should match by construction |
|
|
| 3 | Step 2 lands but a third party reverts `continue-on-error` later | Add a 9th check to `forbidden-patterns` in `ci.yml`: `grep -n 'continue-on-error' .github/workflows/visual-diff.yml` should return zero |
|
|
|
|
## Non-goals
|
|
|
|
- Adding more visual baselines (login page, dashboard, etc.) — out of
|
|
scope. The single homepage baseline is the smoke test of the
|
|
visual-diff pipeline; deeper coverage is per-feature work.
|
|
- Switching to a hosted visual-regression service (Percy, Chromatic,
|
|
Argos) — handles the platform problem cleanly but adds a paid
|
|
dependency. Local + axiom is free and works.
|
|
|
|
## Acceptance
|
|
|
|
- `visual-diff.yml` gates merge (failed diff = red required check).
|
|
- Baseline regenerable via a reviewable Git-native workflow (no SSH
|
|
required, no Mac-overwrite-Linux footgun).
|
|
- `AGENTS.md` + `tests/visual/homepage.spec.ts` docblock no longer
|
|
describe the gate as advisory.
|
|
- Optional 9th forbidden-patterns check locks the gate in place.
|