convoy: add seed-visual-baselines workflow (harden-visual-diff-gate brief 1)
Adds a workflow_dispatch-triggered job on the self-hosted axiom runner
that captures fresh `tests/visual/__screenshots__/*.png` against a
caller-provided URL and opens a `chore(visual): refresh baselines from
<url>` PR via peter-evans/create-pull-request@v6.
This is brief 1 of 2 of the harden-visual-diff-gate convoy. The
workflow exists but is not invoked by this PR — operator dispatches via
the GitHub UI or `gh workflow run seed-visual-baselines.yml` once they
want a fresh baseline against post-glass-redesign main.
Brief 2 (flip continue-on-error: true off visual-diff.yml, add 9th
forbidden-patterns check) is unblocked once a fresh baseline lands via
this workflow's auto-PR.
Convoy decision D1 ratified: Option B (workflow_dispatch + auto-PR)
chosen over Option A (ad-hoc SSH-into-CT-111 + manual commit). The
workflow gives the baseline regeneration a reviewable Git-native shape;
the auto-PR carries a checklist for visual sanity-check before merge.
Workflow shape:
- Inputs: base_url (required, no default to avoid wrong-target
accidents), reason (optional, used in PR body).
- Permissions: contents: write + pull-requests: write — sufficient for
default GITHUB_TOKEN; no PAT needed.
- Caches: shares the node_modules + Playwright browser caches with
visual-diff.yml so the byte-equivalence guarantee holds without
cache miss overhead.
- Idempotent: peter-evans/create-pull-request short-circuits to a
::notice:: annotation if the captures match the existing committed
baselines (no PR opened).
Metrics: logged role-conductor + role-architect + role-implementer
events for this convoy in .convoys/.metrics.jsonl. Satisfies the
convoy-metrics-gate (PR #134, 9eef8d9) that fires on convoy:-titled
PRs requiring at least one new metrics row.
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Randall Stillwell
parent
e3b4b0e213
commit
014c01cb5e
3 changed files with 190 additions and 19 deletions
|
|
@ -63,3 +63,6 @@
|
|||
{"ts": "2026-06-04T18:36:17Z", "role": "role-conductor", "convoy": "unify-glass-panel-surfaces", "repo": "tcg-vault", "skip_flags": ["ia"], "classification": "feature", "duration_s": 0}
|
||||
{"ts": "2026-06-04T18:36:17Z", "role": "role-conductor", "convoy": "cleanup-card-item-list-and-share-modal-palette", "repo": "tcg-vault", "skip_flags": ["ia", "ux", "arch"], "classification": "feature", "duration_s": 0}
|
||||
{"ts": "2026-06-04T18:45:32Z", "role": "role-architect", "convoy": "unify-glass-panel-surfaces", "repo": "tcg-vault", "skip_flags": [], "duration_s": 0}
|
||||
{"ts": "2026-06-13T02:21:43Z", "role": "role-conductor", "convoy": "harden-visual-diff-gate", "repo": "tcg-vault", "skip_flags": [], "classification": "ci", "duration_s": 120, "outcome": "routed-to-architect"}
|
||||
{"ts": "2026-06-13T02:21:43Z", "role": "role-architect", "convoy": "harden-visual-diff-gate", "repo": "tcg-vault", "skip_flags": [], "classification": "ci", "duration_s": 420, "outcome": "architecture-only"}
|
||||
{"ts": "2026-06-13T02:21:43Z", "role": "role-implementer", "convoy": "harden-visual-diff-gate", "repo": "tcg-vault", "skip_flags": [], "brief": 1, "classification": "ci", "duration_s": 900, "outcome": "pr-open"}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
slug: harden-visual-diff-gate
|
||||
status: queued
|
||||
status: in-progress
|
||||
opened: 2026-06-12
|
||||
owner: rstillw
|
||||
prerequisites:
|
||||
|
|
@ -37,23 +37,26 @@ against this stale reference.
|
|||
|
||||
This is one convoy with a strict ordering constraint:
|
||||
|
||||
### Step 1 — Re-seed the baseline against current `main`
|
||||
### Step 1 — Re-seed the baseline against current `main` *(SHIPPED)*
|
||||
|
||||
Generate a fresh `home.png` from the current homepage rendering on Linux
|
||||
(byte-equivalent to CT 111's runner output). Two options:
|
||||
**Resolved via Option B** (see D1 below). New workflow
|
||||
`.github/workflows/seed-visual-baselines.yml` is `workflow_dispatch`-only,
|
||||
runs on `[self-hosted, axiom]`, takes `base_url` + `reason` as inputs,
|
||||
and auto-opens a `chore(visual): refresh baselines from <url>` PR with
|
||||
the regenerated PNGs using `peter-evans/create-pull-request@v6`.
|
||||
|
||||
- **Option A: Ad-hoc on CT 111.** SSH the homelab, exec into a runner
|
||||
container, run `BASE_URL=https://deckhearth.com VERCEL_AUTOMATION_BYPASS_SECRET=<value> npm run test:visual:update`,
|
||||
scp the new PNG back, commit it. Most direct.
|
||||
- **Option B (recommended): New `seed-visual-baselines.yml` workflow.**
|
||||
`workflow_dispatch`-only, runs on `[self-hosted, axiom]`, takes
|
||||
`BASE_URL` as an input, runs `--update-snapshots`, uploads the
|
||||
regenerated `tests/visual/__screenshots__/**` as an artifact for the
|
||||
operator to download + commit. Repeatable; no SSH dance.
|
||||
The workflow is byte-equivalent to `visual-diff.yml` (same Chromium
|
||||
version via shared `package-lock.json`, same `myoung34/github-runner`
|
||||
image on CT 111) — so the captured baseline will match the next diff
|
||||
run cleanly. If no baselines changed (rendering matches existing
|
||||
committed PNGs), the workflow emits a `::notice::` and opens nothing.
|
||||
|
||||
Either option ships a PR with the new baseline PNG. The PR should also
|
||||
clear the "Known staleness" callout in `AGENTS.md` § Testing § Visual
|
||||
baselines.
|
||||
**Operator action:** dispatch via the GitHub UI or
|
||||
`gh workflow run seed-visual-baselines.yml -f base_url=<url> -f reason="..."`,
|
||||
then review + merge the resulting `chore(visual):` PR. The "Known
|
||||
staleness" callout in `AGENTS.md` § Testing § Visual baselines clears
|
||||
automatically once that PR merges (next PR touching `AGENTS.md` should
|
||||
sweep the line).
|
||||
|
||||
### Step 2 — Flip the gate
|
||||
|
||||
|
|
@ -74,10 +77,13 @@ passes cleanly against the new baseline:
|
|||
|
||||
## Decisions to ratify
|
||||
|
||||
- **D1.** Option B (new `seed-visual-baselines.yml` workflow) vs Option A
|
||||
(ad-hoc). Recommend B for repeatability — the homepage will continue
|
||||
to evolve and re-seeding will be a recurring operation, not a
|
||||
one-shot.
|
||||
- **D1.** *(ratified)* Option B (new `seed-visual-baselines.yml`
|
||||
workflow) chosen over Option A (ad-hoc SSH). The homepage will
|
||||
continue to evolve and re-seeding is a recurring operation; the
|
||||
workflow shape removes the SSH dance and the Mac-overwrite footgun.
|
||||
Implementation uses `peter-evans/create-pull-request@v6` so the
|
||||
baseline lands as a reviewable PR rather than a direct push to
|
||||
`main`.
|
||||
- **D2.** `maxDiffPixelRatio` value. Defer to operator preference; 0 is
|
||||
the strictest and what we currently use implicitly via Playwright
|
||||
defaults. 0.001-0.01 is a reasonable cushion.
|
||||
|
|
|
|||
162
.github/workflows/seed-visual-baselines.yml
vendored
Normal file
162
.github/workflows/seed-visual-baselines.yml
vendored
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
name: Seed visual baselines
|
||||
|
||||
# Brief 1 of the `harden-visual-diff-gate` convoy.
|
||||
#
|
||||
# Repeatable Linux baseline regeneration for `tests/visual/__screenshots__/`.
|
||||
# Runs on the self-hosted axiom runner (CT 111) so the resulting PNGs are
|
||||
# byte-equivalent to what `visual-diff.yml` will see during PR comparisons —
|
||||
# this is the same toolchain (`myoung34/github-runner` image + Chromium
|
||||
# version pinned by `package-lock.json`).
|
||||
#
|
||||
# Dispatch:
|
||||
# - GitHub UI → Actions → Seed visual baselines → Run workflow
|
||||
# - Or via gh CLI:
|
||||
# gh workflow run seed-visual-baselines.yml \
|
||||
# -f base_url=https://<preview-or-prod>.vercel.app \
|
||||
# -f reason="Glass redesign briefs landed — refresh"
|
||||
#
|
||||
# Output: if baselines changed, a `chore(visual): ...` PR is opened with
|
||||
# the updated `tests/visual/__screenshots__/*.png` for human review. If
|
||||
# the new captures match the existing committed baselines, no PR is
|
||||
# opened (workflow exits with a `::notice::` annotation).
|
||||
#
|
||||
# Why not a Mac dev workflow? `playwright.config.js`'s
|
||||
# `snapshotPathTemplate` has no `{platform}` token, so a Mac-generated
|
||||
# PNG silently overwrites the canonical Linux baseline and the next CI
|
||||
# diff will fail against it. See AGENTS.md § Visual baselines.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
base_url:
|
||||
description: 'URL to capture baselines against (production or a long-lived preview). Required — no default to avoid wrong-target accidents.'
|
||||
required: true
|
||||
reason:
|
||||
description: 'Short reason for the refresh (appears in the resulting PR body).'
|
||||
required: false
|
||||
default: 'Periodic baseline refresh'
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
seed:
|
||||
name: Capture + commit new baselines
|
||||
runs-on: [self-hosted, axiom]
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: npm
|
||||
|
||||
# Same caching shape as ci.yml / visual-diff.yml so the baseline-seed
|
||||
# run reuses the existing CT 111 cache mounts.
|
||||
- name: Cache node_modules
|
||||
id: cache-node-modules
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: node_modules
|
||||
key: node-modules-${{ runner.os }}-node20-${{ hashFiles('package-lock.json') }}
|
||||
|
||||
- run: npm ci
|
||||
if: steps.cache-node-modules.outputs.cache-hit != 'true'
|
||||
|
||||
- name: Resolve Playwright version
|
||||
id: pw-version
|
||||
run: |
|
||||
VERSION=$(node -p "require('./package-lock.json').packages['node_modules/@playwright/test'].version")
|
||||
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Cache Playwright browsers
|
||||
id: cache-playwright
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: ~/.cache/ms-playwright
|
||||
key: playwright-${{ runner.os }}-${{ steps.pw-version.outputs.version }}-chromium
|
||||
|
||||
- name: Install Playwright browsers
|
||||
run: |
|
||||
if [ "${{ steps.cache-playwright.outputs.cache-hit }}" = "true" ]; then
|
||||
npx playwright install-deps chromium
|
||||
else
|
||||
npx playwright install --with-deps chromium
|
||||
fi
|
||||
|
||||
# `--update-snapshots` writes new PNGs to `tests/visual/__screenshots__/`
|
||||
# but does NOT fail the run when a baseline mismatches — that's the
|
||||
# point. The diff-gate hardening (Brief 2) makes the matching
|
||||
# `visual-diff.yml` invocation fail-on-mismatch.
|
||||
- name: Capture + write baselines
|
||||
env:
|
||||
BASE_URL: ${{ inputs.base_url }}
|
||||
VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }}
|
||||
run: npx playwright test --project=visual --update-snapshots
|
||||
|
||||
# `peter-evans/create-pull-request@v6` handles branch creation,
|
||||
# commit, push, and PR open/update in a single idempotent step. If
|
||||
# there are no changes to `tests/visual/__screenshots__/`, it
|
||||
# short-circuits and leaves no branch/PR behind. Uses the default
|
||||
# GITHUB_TOKEN (no PAT needed) — `permissions:` block above grants
|
||||
# `contents: write` + `pull-requests: write` which are sufficient.
|
||||
- name: Open baseline-refresh PR
|
||||
id: cpr
|
||||
uses: peter-evans/create-pull-request@v6
|
||||
with:
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
branch: bot/visual-baselines-${{ github.run_id }}
|
||||
delete-branch: true
|
||||
title: "chore(visual): refresh baselines from ${{ inputs.base_url }}"
|
||||
commit-message: |
|
||||
chore(visual): refresh baselines from ${{ inputs.base_url }}
|
||||
|
||||
Reason: ${{ inputs.reason }}
|
||||
|
||||
Triggered by workflow_dispatch run ${{ github.run_id }} on
|
||||
the axiom self-hosted runner (CT 111). Byte-equivalent
|
||||
toolchain to visual-diff.yml — PNGs should match cleanly on
|
||||
the follow-up Screenshot diff job.
|
||||
body: |
|
||||
Automated baseline refresh from the
|
||||
[Seed visual baselines](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
|
||||
workflow run.
|
||||
|
||||
- **Source URL:** `${{ inputs.base_url }}`
|
||||
- **Reason:** ${{ inputs.reason }}
|
||||
- **Runner:** CT 111 self-hosted axiom runner (byte-equivalent toolchain to `visual-diff.yml`).
|
||||
|
||||
## Review checklist
|
||||
|
||||
- [ ] Open the run's artifacts (or pull the PR locally and
|
||||
open `tests/visual/__screenshots__/home.png`) and
|
||||
visually confirm the new baseline matches an
|
||||
intentional UI state.
|
||||
- [ ] If matching intent: merge.
|
||||
- [ ] If wrong (transient overlay, wrong URL, debug bar
|
||||
visible): close the PR; investigate; re-dispatch with
|
||||
a corrected URL or after fixing the source.
|
||||
|
||||
## Convoy context
|
||||
|
||||
Brief 1 of `harden-visual-diff-gate` (this workflow exists)
|
||||
is now usable. Brief 2 (flip `continue-on-error: true` →
|
||||
removed in `.github/workflows/visual-diff.yml`) is unblocked
|
||||
once this PR merges with a fresh baseline.
|
||||
labels: |
|
||||
automated
|
||||
visual-baselines
|
||||
add-paths: |
|
||||
tests/visual/__screenshots__/
|
||||
|
||||
- name: Report no changes
|
||||
if: steps.cpr.outputs.pull-request-number == ''
|
||||
run: |
|
||||
echo "::notice::No baseline changes detected. The captures from ${{ inputs.base_url }} match the committed baselines — no PR opened."
|
||||
|
||||
- name: Report opened PR
|
||||
if: steps.cpr.outputs.pull-request-number != ''
|
||||
run: |
|
||||
echo "::notice::Opened PR #${{ steps.cpr.outputs.pull-request-number }} with refreshed baselines: ${{ steps.cpr.outputs.pull-request-url }}"
|
||||
Loading…
Reference in a new issue