rstillwell/deckhearth
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions
deckhearth/.convoys/harden-visual-diff-gate.md
varutasu e3b4b0e213
chore(docs): sweep stale visual-baseline references + queue gate-hardening convoy (#135) 
PR #58 (83a358b, 2026-06-02) committed the first Linux visual baseline,
resolving the seed-visual-baselines-on-linux convoy. But the cleanup
sweep across docs that referenced the convoy as "queued / not yet done"
never landed. Three files still describe the world as if PR #58 hadn't
happened, which confuses any agent reading the codebase to understand
the visual-diff pipeline:

  1. tests/visual/homepage.spec.ts module docblock — described "FIRST
     RUN (no committed baseline yet)" and "SEEDING THE BASELINE
     (post-merge follow-up)" as the active state.
  2. playwright.config.js snapshotPathTemplate comment — said
     "Per Decision 4, we don't commit baselines this convoy" and
     pointed at the queued seed convoy as future work.
  3. AGENTS.md § Testing § Visual baselines + § CI behavior §
     Screenshot diff — claimed "none committed yet" and that the first
     visual-diff run "will fail at the test step because no baseline
     exists yet."

Sweeping all three to describe the current reality. The Mac-vs-Linux
platform footgun (snapshotPathTemplate has no {platform} token) is
still live, so that warning stays — just rephrased from "don't have
baselines yet" to "don't regenerate them on a Mac."

Also surfaces a separate finding the seed work left behind:
visual-diff.yml's screenshot capture step still carries
`continue-on-error: true`, making the diff advisory rather than gating.
Flipping it requires re-seeding the baseline against post-glass-redesign
main first (the PR #58 baseline predates unify-glass-panel-surfaces +
cleanup-card-item-list-and-share-modal-palette +
migrate-button-input-mobilenav-to-glass-primitive). New convoy seed at
.convoys/harden-visual-diff-gate.md captures the two-step shape (re-seed
baseline, then flip the gate) plus the recommended workflow_dispatch
approach for repeatable re-seeding on CT 111.

No code behavior changes. Documentation + .md convoy seed only.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-12 20:30:34 -05:00

5.4 KiB
Raw Blame History

slug: harden-visual-diff-gate status: queued opened: 2026-06-12 owner: rstillw prerequisites:

  • Stable main post-unify-glass-panel-surfaces / cleanup-card-item-list-and-share-modal-palette / migrate-button-input-mobilenav-to-glass-primitive (all merged)
  • CT 111 self-hosted runner online (resolved by PR #132, 2026-06-07) related:
  • PR #58 (83a358b) — initial Linux baseline seeded
  • PR #18 (7b6f751) — adopt-playwright-smoke Decision 4: continue-on-error end state
  • tests/visual/homepage.spec.ts module docblock
  • .github/workflows/visual-diff.yml line ~126

harden-visual-diff-gate

Problem

.github/workflows/visual-diff.yml still carries continue-on-error: true on the "Capture screenshots (PR)" step. This was the documented end state of adopt-playwright-smoke (Decision 4) because no Linux baseline existed yet. PR #58 seeded the baseline on 2026-06-02, but the gate was never flipped — visual drift continues to ship as artifacts + PR comments only, not as a blocking check.

Additionally, the committed baseline (tests/visual/__screenshots__/home.png) was generated against main at 83a358b, which is several convoys behind the current homepage rendering (glass redesign briefs, corner-light tone adjustments, card-class retirement, etc.). Flipping continue-on-error today would fail every PR touching pages/** / components/** / styles/** against this stale reference.

Two-step shape

This is one convoy with a strict ordering constraint:

Step 1 — Re-seed the baseline against current main

Generate a fresh home.png from the current homepage rendering on Linux (byte-equivalent to CT 111's runner output). Two options:

  • Option A: Ad-hoc on CT 111. SSH the homelab, exec into a runner container, run BASE_URL=https://deckhearth.com VERCEL_AUTOMATION_BYPASS_SECRET=<value> npm run test:visual:update, scp the new PNG back, commit it. Most direct.
  • Option B (recommended): New seed-visual-baselines.yml workflow. workflow_dispatch-only, runs on [self-hosted, axiom], takes BASE_URL as an input, runs --update-snapshots, uploads the regenerated tests/visual/__screenshots__/** as an artifact for the operator to download + commit. Repeatable; no SSH dance.

Either option ships a PR with the new baseline PNG. The PR should also clear the "Known staleness" callout in AGENTS.md § Testing § Visual baselines.

Step 2 — Flip the gate

After Step 1's PR merges and a follow-up PR confirms Screenshot diff passes cleanly against the new baseline:

  • Remove continue-on-error: true from .github/workflows/visual-diff.yml (line ~126).
  • Update AGENTS.md § CI behavior § Screenshot diff to drop the "advisory, not gating" language.
  • Update the module docblock in tests/visual/homepage.spec.ts to drop the "still carries continue-on-error: true" callout.
  • Optional: bump expect.toHaveScreenshot.maxDiffPixelRatio in playwright.config.js if some pixel jitter is unavoidable (e.g. font subpixel rendering across runner image bumps). Default is 0 (any pixel diff fails); 0.01 (1%) is a common safe floor for Chromium-vs-Chromium runs.

Decisions to ratify

  • D1. Option B (new seed-visual-baselines.yml workflow) vs Option A (ad-hoc). Recommend B for repeatability — the homepage will continue to evolve and re-seeding will be a recurring operation, not a one-shot.
  • D2. maxDiffPixelRatio value. Defer to operator preference; 0 is the strictest and what we currently use implicitly via Playwright defaults. 0.001-0.01 is a reasonable cushion.
  • D3. Do we also harden preview-smoke.yml? It already runs as a blocking gate (no continue-on-error); no change needed. This convoy is scoped to visual-diff.yml only.

Risks

# Risk Mitigation
1 Re-seeded baseline drifts on next runner-image update Watchtower's weekly update cycle (CT 111) could shift Chromium subpixel rendering. Mitigate via D2's maxDiffPixelRatio cushion
2 Step 1's baseline PR fails its own Screenshot diff Expected — the new baseline IS the visual change. Use pipeline: skip visual directive in the PR body OR open the baseline PR with the skip-metrics label equivalent for visual-diff (currently none — would need a new bypass mechanism). Cleaner: land the baseline PR via the tests/visual/** path filter, which DOES trigger visual-diff but the new screenshot vs new baseline should match by construction
3 Step 2 lands but a third party reverts continue-on-error later Add a 9th check to forbidden-patterns in ci.yml: grep -n 'continue-on-error' .github/workflows/visual-diff.yml should return zero

Non-goals

  • Adding more visual baselines (login page, dashboard, etc.) — out of scope. The single homepage baseline is the smoke test of the visual-diff pipeline; deeper coverage is per-feature work.
  • Switching to a hosted visual-regression service (Percy, Chromatic, Argos) — handles the platform problem cleanly but adds a paid dependency. Local + axiom is free and works.

Acceptance

  • visual-diff.yml gates merge (failed diff = red required check).
  • Baseline regenerable via a reviewable Git-native workflow (no SSH required, no Mac-overwrite-Linux footgun).
  • AGENTS.md + tests/visual/homepage.spec.ts docblock no longer describe the gate as advisory.
  • Optional 9th forbidden-patterns check locks the gate in place.