feat(test): adopt @playwright/test + ship playwright.config.js + visual scaffold (Brief 1 of adopt-playwright-smoke)
Closes P1 #10 step 2 / launch sequence step 10. PR #17 already plumbed VERCEL_AUTOMATION_BYPASS_SECRET into both workflows' env; this brief wires the actual @playwright/test dep, the playwright.config.js that bridges env -> use.extraHTTPHeaders, and a single visual spec so the screenshot workflow has something to discover. Per architect Decision 1 - tests/smoke/app.smoke.spec.ts stays as .ts (npx eslint exits 0 against the current config); new visual spec also .ts. Per Decision 2 - playwright.config.js fail-loud in CI when VERCEL_AUTOMATION_BYPASS_SECRET is missing (throw at config load with actionable message naming the env var, rotation command, AGENTS.md reference); warn-and-no-op in dev. Mirrors lib/rate-limit.js fail-closed pattern (AGENTS.md Gotcha #12). Per Decision 3 - two projects partitioned by testMatch: - smoke -> tests/smoke/**/*.spec.{ts,js} - visual -> tests/visual/**/*.spec.{ts,js} Shared use: block. Matches the workflows' --project=smoke|visual invocations. Per Decision 4 - NO baselines committed. The Mac-vs-Linux platform- suffix issue (architect Boot-the-brief Finding 7: Playwright's custom snapshotPathTemplate has no {platform} token) means a Mac-generated baseline would silently overwrite a Linux CI baseline. Queued as seed-visual-baselines-on-linux follow-up convoy. First CI run of Screenshot diff will fail at the test step; existing continue-on-error: true swallows it and the comment-on-PR step posts the run link. Per Decision 5 - no eslint.config.mjs change. Per Decision 6 - three simple scripts (test:smoke, test:visual, test:visual:update). No auto-boot wrapper. Verification: - npm ci --dry-run exits 0 (lockfile in sync) - npm run lint: 128 problems (baseline match - 81 errors, 47 warnings) - npm run test:run: 21/21 vitest pass (no regression) - npm run test:smoke (no BASE_URL): dev-path warn-and-continue fires "[playwright.config] VERCEL_AUTOMATION_BYPASS_SECRET unset" warning, enumerates 3 smoke tests, attempts to run (fails on missing local chromium binary - workflows install browsers via npx playwright install --with-deps chromium) - CI=true npm run test:smoke: CI-path throws at config load with "VERCEL_AUTOMATION_BYPASS_SECRET is required in CI to reach Vercel-Protection-protected preview deployments. Reseed via: gh secret set VERCEL_AUTOMATION_BYPASS_SECRET --body \"<value>\". See AGENTS.md \xc2\xa7 7 for the full plumbing context." - npx playwright --version: Version 1.60.0 Implementation note: removed the architect's defensive eslint-disable- next-line no-console directive in the dev-warn branch because the active eslint config does not enable no-console for repo-root config files; ESLint reported the directive as unused, which would have regressed the lint baseline by +1. Behavior unchanged; comment-only edit per the brief's explicit "comment wording can be tightened" allowance. Replaced with an explanatory comment noting why the directive is intentionally absent. Implementation note 2: alphabetized @playwright/test as the FIRST entry in devDependencies (before @testing-library/dom) because "@p" < "@t". The brief's prose said "between @testing-library/react and autoprefixer" which would have broken alphabetization; followed the brief's overarching principle ("alphabetical position") instead. No code paths in pages/** or lib/** touched. No workflow YAML edited. No AGENTS.md edits (doc-writer pass at convoy close handles the seed-on-Linux runbook). Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Randall Stillwell
parent
3ac527ef48
commit
c72d006666
5 changed files with 246 additions and 1 deletions
7
.gitignore
vendored
7
.gitignore
vendored
|
|
@ -36,3 +36,10 @@ __pycache__/
|
||||||
|
|
||||||
# Local code-knowledge-graph (per-developer; user-code-review-graph MCP)
|
# Local code-knowledge-graph (per-developer; user-code-review-graph MCP)
|
||||||
.code-review-graph/
|
.code-review-graph/
|
||||||
|
|
||||||
|
# Playwright test runner artifacts (generated on every local run;
|
||||||
|
# never committed). Baselines under `tests/visual/__screenshots__/`
|
||||||
|
# are EXPLICITLY NOT ignored — they must be committed when they exist.
|
||||||
|
/playwright-report/
|
||||||
|
/test-results/
|
||||||
|
/.playwright/
|
||||||
|
|
|
||||||
64
package-lock.json
generated
64
package-lock.json
generated
|
|
@ -23,6 +23,7 @@
|
||||||
"resend": "^4.7.0"
|
"resend": "^4.7.0"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
|
"@playwright/test": "^1.60.0",
|
||||||
"@testing-library/dom": "^10.4.1",
|
"@testing-library/dom": "^10.4.1",
|
||||||
"@testing-library/react": "^16.3.2",
|
"@testing-library/react": "^16.3.2",
|
||||||
"autoprefixer": "^10.4.21",
|
"autoprefixer": "^10.4.21",
|
||||||
|
|
@ -2071,6 +2072,22 @@
|
||||||
"node": ">=14"
|
"node": ">=14"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"node_modules/@playwright/test": {
|
||||||
|
"version": "1.60.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.60.0.tgz",
|
||||||
|
"integrity": "sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"dependencies": {
|
||||||
|
"playwright": "1.60.0"
|
||||||
|
},
|
||||||
|
"bin": {
|
||||||
|
"playwright": "cli.js"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
"node_modules/@react-email/render": {
|
"node_modules/@react-email/render": {
|
||||||
"version": "1.1.2",
|
"version": "1.1.2",
|
||||||
"resolved": "https://registry.npmjs.org/@react-email/render/-/render-1.1.2.tgz",
|
"resolved": "https://registry.npmjs.org/@react-email/render/-/render-1.1.2.tgz",
|
||||||
|
|
@ -7531,6 +7548,53 @@
|
||||||
"node": ">= 6"
|
"node": ">= 6"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"node_modules/playwright": {
|
||||||
|
"version": "1.60.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.60.0.tgz",
|
||||||
|
"integrity": "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"dependencies": {
|
||||||
|
"playwright-core": "1.60.0"
|
||||||
|
},
|
||||||
|
"bin": {
|
||||||
|
"playwright": "cli.js"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
},
|
||||||
|
"optionalDependencies": {
|
||||||
|
"fsevents": "2.3.2"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/playwright-core": {
|
||||||
|
"version": "1.60.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.60.0.tgz",
|
||||||
|
"integrity": "sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"bin": {
|
||||||
|
"playwright-core": "cli.js"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/playwright/node_modules/fsevents": {
|
||||||
|
"version": "2.3.2",
|
||||||
|
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
|
||||||
|
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
|
||||||
|
"dev": true,
|
||||||
|
"hasInstallScript": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"darwin"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
"node_modules/possible-typed-array-names": {
|
"node_modules/possible-typed-array-names": {
|
||||||
"version": "1.1.0",
|
"version": "1.1.0",
|
||||||
"resolved": "https://registry.npmjs.org/possible-typed-array-names/-/possible-typed-array-names-1.1.0.tgz",
|
"resolved": "https://registry.npmjs.org/possible-typed-array-names/-/possible-typed-array-names-1.1.0.tgz",
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,10 @@
|
||||||
"import-popular": "node scripts/import-popular-sets.js",
|
"import-popular": "node scripts/import-popular-sets.js",
|
||||||
"import-all": "node scripts/bulk-import-all.js",
|
"import-all": "node scripts/bulk-import-all.js",
|
||||||
"test": "vitest",
|
"test": "vitest",
|
||||||
"test:run": "vitest run"
|
"test:run": "vitest run",
|
||||||
|
"test:smoke": "playwright test --project=smoke",
|
||||||
|
"test:visual": "playwright test --project=visual",
|
||||||
|
"test:visual:update": "playwright test --project=visual --update-snapshots"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@neondatabase/serverless": "^1.0.1",
|
"@neondatabase/serverless": "^1.0.1",
|
||||||
|
|
@ -30,6 +33,7 @@
|
||||||
"resend": "^4.7.0"
|
"resend": "^4.7.0"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
|
"@playwright/test": "^1.60.0",
|
||||||
"@testing-library/dom": "^10.4.1",
|
"@testing-library/dom": "^10.4.1",
|
||||||
"@testing-library/react": "^16.3.2",
|
"@testing-library/react": "^16.3.2",
|
||||||
"autoprefixer": "^10.4.21",
|
"autoprefixer": "^10.4.21",
|
||||||
|
|
|
||||||
133
playwright.config.js
Normal file
133
playwright.config.js
Normal file
|
|
@ -0,0 +1,133 @@
|
||||||
|
// Playwright config for the post-PR-17 `Preview smoke` and
|
||||||
|
// `Screenshot diff` workflows. ESM per the repo's
|
||||||
|
// `"type": "module"` setting in package.json. Companion docs:
|
||||||
|
// `.convoys/adopt-playwright-smoke.md` (Decisions 1-6),
|
||||||
|
// AGENTS.md § 7 (Vercel preview bypass conventions).
|
||||||
|
|
||||||
|
import { defineConfig } from '@playwright/test';
|
||||||
|
|
||||||
|
const BASE_URL = process.env.BASE_URL ?? 'http://localhost:3000';
|
||||||
|
const BYPASS_SECRET = process.env.VERCEL_AUTOMATION_BYPASS_SECRET;
|
||||||
|
|
||||||
|
// `CI === 'true'` is the canonical GitHub Actions signal (set by
|
||||||
|
// the runner). Playwright config has no Next.js context, so
|
||||||
|
// `NODE_ENV` is not reliable here. Mirrors the rate-limit.js
|
||||||
|
// fail-closed pattern documented in AGENTS.md Gotcha #12.
|
||||||
|
const IS_CI = process.env.CI === 'true';
|
||||||
|
|
||||||
|
if (IS_CI && !BYPASS_SECRET) {
|
||||||
|
// Fail loud in CI per Decision 2. The workflow's `env:` block
|
||||||
|
// (preview-smoke.yml line 101, visual-diff.yml line 85) maps
|
||||||
|
// `secrets.VERCEL_AUTOMATION_BYPASS_SECRET` into the process
|
||||||
|
// env; if it's empty here, the secret is unseeded or the
|
||||||
|
// workflow YAML drift broke the mapping.
|
||||||
|
throw new Error(
|
||||||
|
'VERCEL_AUTOMATION_BYPASS_SECRET is required in CI to reach ' +
|
||||||
|
'Vercel-Protection-protected preview deployments. ' +
|
||||||
|
'Reseed via: gh secret set VERCEL_AUTOMATION_BYPASS_SECRET --body "<value>". ' +
|
||||||
|
'See AGENTS.md § 7 for the full plumbing context.'
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!BYPASS_SECRET && !IS_CI) {
|
||||||
|
// Dev fallback per Decision 2 — warn once at config load,
|
||||||
|
// proceed without the header. Local runs target localhost
|
||||||
|
// (no preview protection) or a non-protected URL.
|
||||||
|
// (No eslint-disable for `no-console` here: the active config
|
||||||
|
// does not enable `no-console` for repo-root config files;
|
||||||
|
// adding the directive here trips `eslint-comments/no-unused-disable`.)
|
||||||
|
console.warn(
|
||||||
|
'[playwright.config] VERCEL_AUTOMATION_BYPASS_SECRET unset — ' +
|
||||||
|
'running without the Vercel bypass header. Targets a non-protected ' +
|
||||||
|
'URL (e.g. http://localhost:3000). Hitting a protected preview without ' +
|
||||||
|
'this header will return Vercel\'s SSO challenge page.'
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
// Both `tests/smoke/` and `tests/visual/` live under `tests/`.
|
||||||
|
// Project-level `testMatch` (below) partitions them so the
|
||||||
|
// two workflows (`--project=smoke` and `--project=visual`)
|
||||||
|
// each see only the specs they should run.
|
||||||
|
testDir: './tests',
|
||||||
|
|
||||||
|
// Smoke + visual specs are independent; parallelism within a
|
||||||
|
// single spec adds no value here and would complicate the
|
||||||
|
// per-test screenshot baseline lifecycle.
|
||||||
|
fullyParallel: false,
|
||||||
|
workers: IS_CI ? 1 : undefined,
|
||||||
|
|
||||||
|
// One retry in CI handles transient Vercel preview flakes
|
||||||
|
// (cold-start, DNS propagation). Local: zero retries — fail
|
||||||
|
// fast so the dev sees the issue immediately.
|
||||||
|
retries: IS_CI ? 1 : 0,
|
||||||
|
|
||||||
|
// 30s per test is plenty for the 3 smoke checks + 1 visual
|
||||||
|
// screenshot. The convoy file's success metric is < 5min
|
||||||
|
// total workflow runtime; per-test 30s is well inside that.
|
||||||
|
timeout: 30_000,
|
||||||
|
expect: { timeout: 10_000 },
|
||||||
|
|
||||||
|
// List reporter in dev for human readability; add HTML in CI
|
||||||
|
// so the `Upload Playwright report on failure` step
|
||||||
|
// (preview-smoke.yml line 104) has a populated `playwright-report/`
|
||||||
|
// to upload. `open: 'never'` keeps the HTML from auto-launching
|
||||||
|
// a browser tab in headless CI.
|
||||||
|
reporter: IS_CI ? [['list'], ['html', { open: 'never' }]] : 'list',
|
||||||
|
|
||||||
|
// Visual baselines live at `tests/visual/__screenshots__/<arg>{ext}`.
|
||||||
|
// Workflow `visual-diff.yml` line 94 uploads this exact path as the
|
||||||
|
// artifact — keep them aligned. {arg} is the snapshot name from
|
||||||
|
// `toHaveScreenshot('home.png')` without the extension; {ext} is
|
||||||
|
// the extension with the leading dot.
|
||||||
|
//
|
||||||
|
// Cross-platform note (Risk R3): this template drops Playwright's
|
||||||
|
// default `-<browser>-<platform>` suffix. That means a Mac dev
|
||||||
|
// running `npm run test:visual:update` overwrites the Linux-CI
|
||||||
|
// baseline. Per Decision 4, we don't commit baselines this convoy;
|
||||||
|
// operator seeds via `mcr.microsoft.com/playwright:v1.60.0-noble`
|
||||||
|
// Docker on Linux. Multi-platform support is the
|
||||||
|
// `seed-visual-baselines-on-linux` follow-up convoy's job.
|
||||||
|
snapshotPathTemplate: 'tests/visual/__screenshots__/{arg}{ext}',
|
||||||
|
|
||||||
|
use: {
|
||||||
|
baseURL: BASE_URL,
|
||||||
|
// Headers apply to BOTH browser `page.goto(...)` calls AND the
|
||||||
|
// test-level `request` fixture's APIRequestContext (verified
|
||||||
|
// against Playwright docs: `testOptions.extraHTTPHeaders` is
|
||||||
|
// shared between browser context and APIRequestContext
|
||||||
|
// construction). This is why `tests/smoke/app.smoke.spec.ts`'s
|
||||||
|
// third test (`request.get('/api/health')`) reaches the
|
||||||
|
// protected preview without re-injecting the header in the
|
||||||
|
// spec body. If a future hotfix shows the header NOT
|
||||||
|
// propagating to APIRequestContext, see Risk R1 in the
|
||||||
|
// convoy file's Architecture section.
|
||||||
|
extraHTTPHeaders: BYPASS_SECRET
|
||||||
|
? { 'x-vercel-protection-bypass': BYPASS_SECRET }
|
||||||
|
: undefined,
|
||||||
|
// Trace OFF this convoy. Enabling it would land the bypass
|
||||||
|
// header in the HAR payload (Risk R6); a future polish convoy
|
||||||
|
// owns the trace-on + HAR-sanitization decision.
|
||||||
|
trace: 'off',
|
||||||
|
screenshot: 'off',
|
||||||
|
video: 'off',
|
||||||
|
},
|
||||||
|
|
||||||
|
projects: [
|
||||||
|
{
|
||||||
|
// `Preview smoke` workflow invokes `--project=smoke` (per
|
||||||
|
// preview-smoke.yml line 98). The testMatch keeps the
|
||||||
|
// visual specs out of this project.
|
||||||
|
name: 'smoke',
|
||||||
|
testMatch: 'smoke/**/*.spec.@(ts|js)',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// `Screenshot diff` workflow invokes `--project=visual`
|
||||||
|
// (per visual-diff.yml line 82). The testMatch keeps the
|
||||||
|
// smoke specs out of this project (so a `--project=visual`
|
||||||
|
// run doesn't redundantly execute the smoke tests).
|
||||||
|
name: 'visual',
|
||||||
|
testMatch: 'visual/**/*.spec.@(ts|js)',
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
37
tests/visual/homepage.spec.ts
Normal file
37
tests/visual/homepage.spec.ts
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
import { test, expect } from '@playwright/test';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Visual baseline for the public homepage.
|
||||||
|
*
|
||||||
|
* FIRST RUN (no committed baseline yet):
|
||||||
|
* The Screenshot diff workflow runs `playwright test --project=visual
|
||||||
|
* --update-snapshots=none` (per .github/workflows/visual-diff.yml). With
|
||||||
|
* no baseline file at `tests/visual/__screenshots__/home.png` AND the
|
||||||
|
* `none` flag, this test FAILS — and that's the documented end state of
|
||||||
|
* the `adopt-playwright-smoke` convoy (Decision 4 in
|
||||||
|
* `.convoys/adopt-playwright-smoke.md`). The workflow's
|
||||||
|
* `continue-on-error: true` swallows the failure and the comment-on-PR
|
||||||
|
* step posts "Visual Diff — view run" with empty artifacts.
|
||||||
|
*
|
||||||
|
* SEEDING THE BASELINE (post-merge follow-up):
|
||||||
|
* Run `npm run test:visual:update` in a Linux environment so the
|
||||||
|
* generated PNG matches what CI will produce. The cleanest path is the
|
||||||
|
* Playwright Docker image:
|
||||||
|
*
|
||||||
|
* docker run --rm -v "$PWD":/work -w /work \
|
||||||
|
* mcr.microsoft.com/playwright:v1.60.0-noble \
|
||||||
|
* sh -c "npm ci && BASE_URL=https://<preview>.vercel.app \
|
||||||
|
* VERCEL_AUTOMATION_BYPASS_SECRET=<value> \
|
||||||
|
* npm run test:visual:update"
|
||||||
|
*
|
||||||
|
* Then commit `tests/visual/__screenshots__/home.png`. This is tracked
|
||||||
|
* as the `seed-visual-baselines-on-linux` follow-up convoy.
|
||||||
|
*/
|
||||||
|
const BASE = process.env.BASE_URL ?? 'http://localhost:3000';
|
||||||
|
|
||||||
|
test.describe('visual: public homepage', () => {
|
||||||
|
test('home renders consistently against baseline', async ({ page }) => {
|
||||||
|
await page.goto(BASE);
|
||||||
|
await expect(page).toHaveScreenshot('home.png');
|
||||||
|
});
|
||||||
|
});
|
||||||
Loading…
Reference in a new issue