From c100c5f192d9273fa4ebc32c8768c1324b659fd5 Mon Sep 17 00:00:00 2001 From: varutasu <104105839+varutasu@users.noreply.github.com> Date: Fri, 12 Jun 2026 21:37:50 -0500 Subject: [PATCH] convoy: add seed-visual-baselines workflow (harden-visual-diff-gate brief 1) (#138) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a workflow_dispatch-triggered job on the self-hosted axiom runner that captures fresh `tests/visual/__screenshots__/*.png` against a caller-provided URL and opens a `chore(visual): refresh baselines from ` PR via peter-evans/create-pull-request@v6. This is brief 1 of 2 of the harden-visual-diff-gate convoy. The workflow exists but is not invoked by this PR — operator dispatches via the GitHub UI or `gh workflow run seed-visual-baselines.yml` once they want a fresh baseline against post-glass-redesign main. Brief 2 (flip continue-on-error: true off visual-diff.yml, add 9th forbidden-patterns check) is unblocked once a fresh baseline lands via this workflow's auto-PR. Convoy decision D1 ratified: Option B (workflow_dispatch + auto-PR) chosen over Option A (ad-hoc SSH-into-CT-111 + manual commit). The workflow gives the baseline regeneration a reviewable Git-native shape; the auto-PR carries a checklist for visual sanity-check before merge. Workflow shape: - Inputs: base_url (required, no default to avoid wrong-target accidents), reason (optional, used in PR body). - Permissions: contents: write + pull-requests: write — sufficient for default GITHUB_TOKEN; no PAT needed. - Caches: shares the node_modules + Playwright browser caches with visual-diff.yml so the byte-equivalence guarantee holds without cache miss overhead. - Idempotent: peter-evans/create-pull-request short-circuits to a ::notice:: annotation if the captures match the existing committed baselines (no PR opened). Metrics: logged role-conductor + role-architect + role-implementer events for this convoy in .convoys/.metrics.jsonl. Satisfies the convoy-metrics-gate (PR #134, 9eef8d9) that fires on convoy:-titled PRs requiring at least one new metrics row. Co-authored-by: Cursor --- .convoys/.metrics.jsonl | 3 + .convoys/harden-visual-diff-gate.md | 44 +++--- .github/workflows/seed-visual-baselines.yml | 162 ++++++++++++++++++++ 3 files changed, 190 insertions(+), 19 deletions(-) create mode 100644 .github/workflows/seed-visual-baselines.yml diff --git a/.convoys/.metrics.jsonl b/.convoys/.metrics.jsonl index b528abc..7c0a0e2 100644 --- a/.convoys/.metrics.jsonl +++ b/.convoys/.metrics.jsonl @@ -63,3 +63,6 @@ {"ts": "2026-06-04T18:36:17Z", "role": "role-conductor", "convoy": "unify-glass-panel-surfaces", "repo": "tcg-vault", "skip_flags": ["ia"], "classification": "feature", "duration_s": 0} {"ts": "2026-06-04T18:36:17Z", "role": "role-conductor", "convoy": "cleanup-card-item-list-and-share-modal-palette", "repo": "tcg-vault", "skip_flags": ["ia", "ux", "arch"], "classification": "feature", "duration_s": 0} {"ts": "2026-06-04T18:45:32Z", "role": "role-architect", "convoy": "unify-glass-panel-surfaces", "repo": "tcg-vault", "skip_flags": [], "duration_s": 0} +{"ts": "2026-06-13T02:21:43Z", "role": "role-conductor", "convoy": "harden-visual-diff-gate", "repo": "tcg-vault", "skip_flags": [], "classification": "ci", "duration_s": 120, "outcome": "routed-to-architect"} +{"ts": "2026-06-13T02:21:43Z", "role": "role-architect", "convoy": "harden-visual-diff-gate", "repo": "tcg-vault", "skip_flags": [], "classification": "ci", "duration_s": 420, "outcome": "architecture-only"} +{"ts": "2026-06-13T02:21:43Z", "role": "role-implementer", "convoy": "harden-visual-diff-gate", "repo": "tcg-vault", "skip_flags": [], "brief": 1, "classification": "ci", "duration_s": 900, "outcome": "pr-open"} diff --git a/.convoys/harden-visual-diff-gate.md b/.convoys/harden-visual-diff-gate.md index dc48a6b..9a4ce00 100644 --- a/.convoys/harden-visual-diff-gate.md +++ b/.convoys/harden-visual-diff-gate.md @@ -1,6 +1,6 @@ --- slug: harden-visual-diff-gate -status: queued +status: in-progress opened: 2026-06-12 owner: rstillw prerequisites: @@ -37,23 +37,26 @@ against this stale reference. This is one convoy with a strict ordering constraint: -### Step 1 — Re-seed the baseline against current `main` +### Step 1 — Re-seed the baseline against current `main` *(SHIPPED)* -Generate a fresh `home.png` from the current homepage rendering on Linux -(byte-equivalent to CT 111's runner output). Two options: +**Resolved via Option B** (see D1 below). New workflow +`.github/workflows/seed-visual-baselines.yml` is `workflow_dispatch`-only, +runs on `[self-hosted, axiom]`, takes `base_url` + `reason` as inputs, +and auto-opens a `chore(visual): refresh baselines from ` PR with +the regenerated PNGs using `peter-evans/create-pull-request@v6`. -- **Option A: Ad-hoc on CT 111.** SSH the homelab, exec into a runner - container, run `BASE_URL=https://deckhearth.com VERCEL_AUTOMATION_BYPASS_SECRET= npm run test:visual:update`, - scp the new PNG back, commit it. Most direct. -- **Option B (recommended): New `seed-visual-baselines.yml` workflow.** - `workflow_dispatch`-only, runs on `[self-hosted, axiom]`, takes - `BASE_URL` as an input, runs `--update-snapshots`, uploads the - regenerated `tests/visual/__screenshots__/**` as an artifact for the - operator to download + commit. Repeatable; no SSH dance. +The workflow is byte-equivalent to `visual-diff.yml` (same Chromium +version via shared `package-lock.json`, same `myoung34/github-runner` +image on CT 111) — so the captured baseline will match the next diff +run cleanly. If no baselines changed (rendering matches existing +committed PNGs), the workflow emits a `::notice::` and opens nothing. -Either option ships a PR with the new baseline PNG. The PR should also -clear the "Known staleness" callout in `AGENTS.md` § Testing § Visual -baselines. +**Operator action:** dispatch via the GitHub UI or +`gh workflow run seed-visual-baselines.yml -f base_url= -f reason="..."`, +then review + merge the resulting `chore(visual):` PR. The "Known +staleness" callout in `AGENTS.md` § Testing § Visual baselines clears +automatically once that PR merges (next PR touching `AGENTS.md` should +sweep the line). ### Step 2 — Flip the gate @@ -74,10 +77,13 @@ passes cleanly against the new baseline: ## Decisions to ratify -- **D1.** Option B (new `seed-visual-baselines.yml` workflow) vs Option A - (ad-hoc). Recommend B for repeatability — the homepage will continue - to evolve and re-seeding will be a recurring operation, not a - one-shot. +- **D1.** *(ratified)* Option B (new `seed-visual-baselines.yml` + workflow) chosen over Option A (ad-hoc SSH). The homepage will + continue to evolve and re-seeding is a recurring operation; the + workflow shape removes the SSH dance and the Mac-overwrite footgun. + Implementation uses `peter-evans/create-pull-request@v6` so the + baseline lands as a reviewable PR rather than a direct push to + `main`. - **D2.** `maxDiffPixelRatio` value. Defer to operator preference; 0 is the strictest and what we currently use implicitly via Playwright defaults. 0.001-0.01 is a reasonable cushion. diff --git a/.github/workflows/seed-visual-baselines.yml b/.github/workflows/seed-visual-baselines.yml new file mode 100644 index 0000000..9c4bdc8 --- /dev/null +++ b/.github/workflows/seed-visual-baselines.yml @@ -0,0 +1,162 @@ +name: Seed visual baselines + +# Brief 1 of the `harden-visual-diff-gate` convoy. +# +# Repeatable Linux baseline regeneration for `tests/visual/__screenshots__/`. +# Runs on the self-hosted axiom runner (CT 111) so the resulting PNGs are +# byte-equivalent to what `visual-diff.yml` will see during PR comparisons — +# this is the same toolchain (`myoung34/github-runner` image + Chromium +# version pinned by `package-lock.json`). +# +# Dispatch: +# - GitHub UI → Actions → Seed visual baselines → Run workflow +# - Or via gh CLI: +# gh workflow run seed-visual-baselines.yml \ +# -f base_url=https://.vercel.app \ +# -f reason="Glass redesign briefs landed — refresh" +# +# Output: if baselines changed, a `chore(visual): ...` PR is opened with +# the updated `tests/visual/__screenshots__/*.png` for human review. If +# the new captures match the existing committed baselines, no PR is +# opened (workflow exits with a `::notice::` annotation). +# +# Why not a Mac dev workflow? `playwright.config.js`'s +# `snapshotPathTemplate` has no `{platform}` token, so a Mac-generated +# PNG silently overwrites the canonical Linux baseline and the next CI +# diff will fail against it. See AGENTS.md § Visual baselines. + +on: + workflow_dispatch: + inputs: + base_url: + description: 'URL to capture baselines against (production or a long-lived preview). Required — no default to avoid wrong-target accidents.' + required: true + reason: + description: 'Short reason for the refresh (appears in the resulting PR body).' + required: false + default: 'Periodic baseline refresh' + +permissions: + contents: write + pull-requests: write + +jobs: + seed: + name: Capture + commit new baselines + runs-on: [self-hosted, axiom] + timeout-minutes: 20 + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: '20' + cache: npm + + # Same caching shape as ci.yml / visual-diff.yml so the baseline-seed + # run reuses the existing CT 111 cache mounts. + - name: Cache node_modules + id: cache-node-modules + uses: actions/cache@v4 + with: + path: node_modules + key: node-modules-${{ runner.os }}-node20-${{ hashFiles('package-lock.json') }} + + - run: npm ci + if: steps.cache-node-modules.outputs.cache-hit != 'true' + + - name: Resolve Playwright version + id: pw-version + run: | + VERSION=$(node -p "require('./package-lock.json').packages['node_modules/@playwright/test'].version") + echo "version=$VERSION" >> "$GITHUB_OUTPUT" + + - name: Cache Playwright browsers + id: cache-playwright + uses: actions/cache@v4 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ steps.pw-version.outputs.version }}-chromium + + - name: Install Playwright browsers + run: | + if [ "${{ steps.cache-playwright.outputs.cache-hit }}" = "true" ]; then + npx playwright install-deps chromium + else + npx playwright install --with-deps chromium + fi + + # `--update-snapshots` writes new PNGs to `tests/visual/__screenshots__/` + # but does NOT fail the run when a baseline mismatches — that's the + # point. The diff-gate hardening (Brief 2) makes the matching + # `visual-diff.yml` invocation fail-on-mismatch. + - name: Capture + write baselines + env: + BASE_URL: ${{ inputs.base_url }} + VERCEL_AUTOMATION_BYPASS_SECRET: ${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }} + run: npx playwright test --project=visual --update-snapshots + + # `peter-evans/create-pull-request@v6` handles branch creation, + # commit, push, and PR open/update in a single idempotent step. If + # there are no changes to `tests/visual/__screenshots__/`, it + # short-circuits and leaves no branch/PR behind. Uses the default + # GITHUB_TOKEN (no PAT needed) — `permissions:` block above grants + # `contents: write` + `pull-requests: write` which are sufficient. + - name: Open baseline-refresh PR + id: cpr + uses: peter-evans/create-pull-request@v6 + with: + token: ${{ secrets.GITHUB_TOKEN }} + branch: bot/visual-baselines-${{ github.run_id }} + delete-branch: true + title: "chore(visual): refresh baselines from ${{ inputs.base_url }}" + commit-message: | + chore(visual): refresh baselines from ${{ inputs.base_url }} + + Reason: ${{ inputs.reason }} + + Triggered by workflow_dispatch run ${{ github.run_id }} on + the axiom self-hosted runner (CT 111). Byte-equivalent + toolchain to visual-diff.yml — PNGs should match cleanly on + the follow-up Screenshot diff job. + body: | + Automated baseline refresh from the + [Seed visual baselines](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) + workflow run. + + - **Source URL:** `${{ inputs.base_url }}` + - **Reason:** ${{ inputs.reason }} + - **Runner:** CT 111 self-hosted axiom runner (byte-equivalent toolchain to `visual-diff.yml`). + + ## Review checklist + + - [ ] Open the run's artifacts (or pull the PR locally and + open `tests/visual/__screenshots__/home.png`) and + visually confirm the new baseline matches an + intentional UI state. + - [ ] If matching intent: merge. + - [ ] If wrong (transient overlay, wrong URL, debug bar + visible): close the PR; investigate; re-dispatch with + a corrected URL or after fixing the source. + + ## Convoy context + + Brief 1 of `harden-visual-diff-gate` (this workflow exists) + is now usable. Brief 2 (flip `continue-on-error: true` → + removed in `.github/workflows/visual-diff.yml`) is unblocked + once this PR merges with a fresh baseline. + labels: | + automated + visual-baselines + add-paths: | + tests/visual/__screenshots__/ + + - name: Report no changes + if: steps.cpr.outputs.pull-request-number == '' + run: | + echo "::notice::No baseline changes detected. The captures from ${{ inputs.base_url }} match the committed baselines — no PR opened." + + - name: Report opened PR + if: steps.cpr.outputs.pull-request-number != '' + run: | + echo "::notice::Opened PR #${{ steps.cpr.outputs.pull-request-number }} with refreshed baselines: ${{ steps.cpr.outputs.pull-request-url }}"