deckhearth/.convoys/migration-tool.md
Randall Stillwell a941d4db1f docs: post-convoy cleanup for 7-convoy 2026-05-26 wave
Updates ship-readiness.md, AGENTS.md, and 7 convoy files to reflect
the as-shipped state of the 2026-05-26 7-convoy multitask wave:

- PR #26 tighten-visual-diff-path-filter (P3)
- PR #27 purge-weak-creds-from-helpers (P2, closes the umbrella)
- PR #28 cleanup-mobile-nav-dead-props (P3)
- PR #29 lint-against-cjs-in-esm-scripts (P3, surfaced by PR #25)
- PR #30 single-sql-client (P1 #8 RESOLVED)
- PR #31 single-auth-provider (P1 #9 RESOLVED)
- PR #32 migration-tool (P1 #11 RESOLVED)

Milestone: 5 of 6 P1 quality items RESOLVED. Only fix-lint-baseline
(P1 #11.5) remains in the P1 lane.

Newly queued follow-ups:
- purge-quick-login-from-loginpage (surfaced by PR #27)
- purge-neondatabase-serverless-fully (surfaced by PR #30,
  unblocked by PR #32's migration tool adoption)

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-26 23:12:26 -05:00

612 lines
33 KiB
Markdown

---
name: migration-tool
classification: quality
priority: P1 (launch sequence step 7)
success_metric: |
Schema changes ship as a migration file under `migrations/` at the
repo root and are applied by `npm run migrate up`. The initial
backfill migration reproduces `scripts/setup-neon-db.js`'s 7-table
bootstrap DDL verbatim. `npm run setup-db` invokes the migration
runner before seeding the admin user. `.convoys/ship-readiness.md`
P1 #11 ("No migration tool — `scripts/add-*.js` graveyard") flips
from open → RESOLVED.
skip:
- role-design-system-auditor
- role-a11y-auditor
- role-ux-reviewer
- role-ia-architect
status: shipped
created: 2026-05-26
shipped: 2026-05-26
parent: ship-readiness
addresses: P1 #11 (launch sequence step 7)
depends_on:
- drop-public-setup (setup-neon-db.js was previously CJS — already ESM post-Brief 2; this convoy assumes ESM)
- fix-reset-db-script (precedent for the ESM/env-var/seed-line shape that setup-neon-db.js now shares)
---
# Convoy: migration-tool
Adopt `node-pg-migrate` as the schema-change tool. Backfill a single
initial migration that reproduces `scripts/setup-neon-db.js`'s
documented bootstrap DDL. Modify `setup-neon-db.js` to invoke the
migration runner before seeding the admin user. Document the new
convention.
## Background — the graveyard problem
`AGENTS.md` Gotcha #6 (pre-convoy) flagged 27+ scripts in `scripts/`
of the form `add-*.js` / `fix-*.js` / `seed-*.js`, each a one-shot
DDL change applied once to prod with no idempotency tracking and no
rollback. Onboarding a fresh env required re-running every script in
the right order; there was no way to know what had been run on a
given Neon branch; every new column was at risk of being missed in
prod.
The lone existing "tool-shaped" migration was
`scripts/migrations/2026-05-24-rename-admin-email.js` (`pick-a-name`
convoy). It uses ESM, is idempotent, and is UNIQUE-collision-safe —
the right shape for a single migration script, but it is NOT wrapped
by any migration tool, so re-running it is the operator's
responsibility and the operator gets no signal about whether the
migration has already been applied to a given env.
The `.cursor/rules/no-go-zones.mdc` rule (pre-convoy) documented the
ad-hoc `scripts/migrations/YYYY-MM-DD-<slug>.js` convention as a
placeholder *"until a real migration tool is adopted"*. This convoy
adopts that tool.
## Decisions
Seven architect decisions. D1-D2 + D4-D5 + D7 are parent-self-ratified
following the convoy spec's recommended choices verbatim (the
recommendations included tuning evidence and matched the project's
existing patterns). D3 + D6 are mechanically determined by D1-D2.
This is a single-implementer convoy; no operator gate was needed
because the spec pre-ratified each Decision's recommended path.
### D1 — Tool: `node-pg-migrate@^8`
**Ratified: `node-pg-migrate@^8.0.4`** (the version that resolves at
install time; pinned to `^8` in `devDependencies`).
Considered alternatives:
| Tool | Why rejected |
| --- | --- |
| `drizzle-kit` | Schema-as-code in TypeScript; would force broader TS adoption than `AGENTS.md` Gotcha #9 allows (TS is a devDep only for the `eslint-config-next` chain). Out of scope. |
| `prisma migrate` | Whole-ORM adoption is a multi-month surface change. Out of scope. |
| `kysely` migrations | TypeScript-first; same TS-adoption issue as drizzle-kit. Out of scope. |
| Hand-rolled in-house tool | Reinvents the tracking-table + locking + dry-run + idempotency-mark surface; pure cost, zero novel value. |
`node-pg-migrate` matches the repo's existing style:
- JavaScript-only (no `tsconfig.json` required; ESM migrations work
out of the box because `package.json` has `"type": "module"`).
- Raw SQL-friendly via `pgm.sql(...)` — no schema-as-code DSL to
learn. The pre-existing `2026-05-24-rename-admin-email.js`
migration is hand-written SQL-shaped JS; the new tool wraps that
exact shape.
- Lightweight: ~533 KB unpacked; only `glob@~11.1.0` + `yargs@~17.7.0`
as direct deps.
- Lowest-magic option in the candidate set: no codegen, no
declarative diff, no opinions about file layout beyond a
conventional `migrations/` folder.
Cost: brings in `pg@^8.21.0` as a peer dep (needed because
`node-pg-migrate` uses the standard TCP `pg` client, not Neon's HTTP
driver). That's an extra ~3 MB of `node_modules`; no runtime cost
because the migration tool is dev-only / scripts-only and never
loads in the Next.js bundle.
### D2 — Migrations directory: `migrations/` at repo root
**Ratified: `migrations/` at the repo root.**
`scripts/migrations/` already exists as the historical placeholder
folder (housing `2026-05-24-rename-admin-email.js`). Per the convoy
spec's suggested split, the new tool-wrapped migrations live in a
fresh top-level `migrations/` directory:
- Separates the tool-wrapped artifacts from the historical
placeholder. The placeholder folder is preserved for the audit
trail of the lone pre-tool migration; agents reading
`scripts/migrations/` see "this is history, NOT how new schema
changes ship".
- Matches `node-pg-migrate`'s default `--migrations-dir migrations`
(zero CLI noise for the common case).
- Keeps tool-managed state out of `scripts/`, which already houses 27
append-only legacy jobs that are flagged as no-go-zones.
### D3 — Tracking table: default `pgmigrations`
**Ratified: accept the default.** No existing table in the schema
collides with `pgmigrations` (verified by reading
`scripts/setup-neon-db.js`'s 7-table DDL and by cross-referencing
`docs/SCHEMA_MAP.md`). Zero CLI flag, zero documentation overhead.
### D4 — Backfill strategy: hand-translate `setup-neon-db.js`
**Ratified: hand-translate `scripts/setup-neon-db.js`'s DDL into the
initial backfill migration.**
Considered alternative: run `pg_dump --schema-only` against a fresh
`npm run setup-db` to capture the bootstrap shape mechanically. The
mechanical approach would be marginally more "guaranteed correct"
versus the hand-translated copy, but requires standing up a
throwaway Neon branch, dotenv-wiring `pg_dump`, sanitizing the
output (stripping owner/permissions lines that `pg_dump` adds), and
manually accepting the resulting normalized SQL into the migration
file. That's more moving parts than a verbatim copy from a single
documented file.
The hand-translation is mechanically simple: each `await sql\`\`` block
in `setup-neon-db.js` becomes one `pgm.sql(\`...\`)` call in the
migration's `up()`. The seven `CREATE TABLE IF NOT EXISTS` blocks
are preserved verbatim (including column order, types, defaults, FK
clauses, and the UNIQUE constraints). The admin-row INSERT in
`setup-neon-db.js` is NOT replicated into the migration — that's
the seed step, which stays in `setup-neon-db.js` (per D5).
**Documented assumption:** `setup-neon-db.js` is the bootstrap
source-of-truth that the project has shipped with since first commit.
If the prod schema has drifted from that bootstrap shape (which it
HAS — the 27 historical `scripts/add-*.js` jobs added many columns
that the bootstrap script never created), the drift is out-of-scope
for this convoy. The backfill migration captures the
`setup-neon-db.js`-shape only; reconciling the full prod shape into
the migration history is the queued
`reconcile-historical-add-scripts` follow-up convoy.
The choice is documented as a known limitation in `docs/SCHEMA_MAP.md`'s
new preamble: *"the initial backfill captures only the
post-`setup-neon-db.js` shape ... if a fresh env needs the full
historical column set, a follow-up convoy ... will fold the historical
effects into the migration history; until then this file remains the
curated reference for the full prod shape."*
The use of `CREATE TABLE IF NOT EXISTS` in the backfill (matching
`setup-neon-db.js` verbatim) means the initial migration is safe to
run against existing envs — every CREATE is a no-op on tables that
already exist (the historical add-*.js columns are preserved). Only
the `pgmigrations` row changes.
### D5 — Bootstrap reconciliation: split DDL (migrations) from seed (`setup-neon-db.js`)
**Ratified: split.** `setup-neon-db.js` now:
1. Validates `ADMIN_INITIAL_PASSWORD` is set (fail loud BEFORE touching the DB).
2. Validates `POSTGRES_URL` is set (new — was previously implicit).
3. Spawns `npm run migrate up` via `node:child_process.spawn` with
`stdio: 'inherit'`. If migrate exits non-zero, setup-db exits non-zero
with a wrapping error message that surfaces the failing exit code +
signal and points the operator at `npm run migrate up` to re-try just
the migration step.
4. Connects to Neon (via the existing `@neondatabase/serverless` HTTP driver)
and runs the admin-row INSERT with `ON CONFLICT (email) DO NOTHING`.
The seven `await sql\`CREATE TABLE IF NOT EXISTS ...\`` blocks are
removed from `setup-neon-db.js` — they now live in
`migrations/1779853647564_initial-schema.js`. The script's success/error
message copy is updated to mention the migration step explicitly so
the operator's mental model matches the new pipeline.
Rejected alternative: keep all DDL in `setup-neon-db.js`, have the
migration runner be a separate `npm run migrate` invocation operators
remember to call. This would leave the DDL-ownership ambiguity intact
(two sources of truth) and require operators to know to call both
scripts in the right order — exactly the kind of operator-burden that
the convoy is trying to remove.
### D6 — CI integration: defer
**Ratified: defer to `wire-migrate-into-ci` follow-up convoy.**
Adding a CI job that runs `npm run migrate up` against a test Neon
branch (or against a temporary Postgres container) requires either:
- A dedicated test Neon branch + a `MIGRATE_TEST_DATABASE_URL` secret
in GitHub Actions, plus branch-reset logic so successive PRs don't
see each other's migrations.
- A Postgres service container in the workflow YAML, which means a
~30s container-start tax on every PR and a CI-specific code path for
the migration runner.
Both are real work. The convoy spec explicitly authorizes deferring
this to a follow-up — surfaced as `wire-migrate-into-ci` in the
follow-ups list below. **Risk acknowledged:** until that follow-up
lands, new migration files are validated at code-review time only
(via `node --check`, the implementer's local `npm run migrate up`
against their dev branch, and any operator-led pre-deploy run).
### D7 — Down-migration shape on the initial backfill: hard stub
**Ratified: hard stub that throws.** Per the convoy spec's recommended
"too risky to drop prod schema" path.
Rolling back the initial schema would drop every users / cards /
collections / decks row in the database — including the
`pgmigrations` row itself. There is no realistic ops scenario where
this is the right thing to do; if a developer genuinely needs a
clean schema for testing, branching the Neon database is instant +
cheap and produces a strictly better outcome (zero data loss for
other developers using the same branch).
The stub's error message names the migration explicitly, explains the
risk, and points at the recommended alternative (Neon branch +
forward-apply). Future migrations that touch one of the seven
bootstrap tables should write their OWN dated migration with a real
`down()` — they do NOT need to re-enable the down on this initial
backfill.
## The fix
| File | Action | Purpose |
| --- | --- | --- |
| `migrations/1779853647564_initial-schema.js` | new | The backfill migration. `up()` runs seven `pgm.sql(\`CREATE TABLE IF NOT EXISTS ...\`)` blocks reproducing `scripts/setup-neon-db.js`'s 7-table DDL verbatim (users / cards / user_cards / collections / collection_cards / decks / deck_cards). `down()` throws (D7). `shorthands` is `undefined`. |
| `package.json` | modified | Adds `"migrate": "node-pg-migrate --database-url-var POSTGRES_URL --envPath .env.local --migrations-dir migrations --verbose"` to `scripts`. Adds `node-pg-migrate@^8.0.4` and `pg@^8.21.0` to `devDependencies`. `setup-db` script unchanged at the YAML layer; its behavior changes per D5. |
| `scripts/setup-neon-db.js` | modified | Splits DDL (now in the migration) from seed (admin-row INSERT). Adds a `POSTGRES_URL` env-var check (was previously implicit). Adds a `runMigrations()` helper that spawns `npm run migrate up` and rejects with a wrapped error on non-zero exit. Updates the success/error message copy to mention the migration step. |
| `README.md` | modified | § Installation step 4 now mentions that `setup-db` chains the migration runner. New § "Schema changes" documents the `npm run migrate create ...` → edit → `npm run migrate up` → commit flow. § "First-time admin setup" mentions the migrate step. |
| `AGENTS.md` | modified | § 3 Conventions: new "Schema changes (post-`migration-tool`)" bullet pointing at the new flow. § 4 Gotchas: Gotcha #6 flipped from open → RESOLVED with the as-shipped paragraph (tool / dir / tracking-table / idempotency notes). |
| `.cursor/rules/no-go-zones.mdc` | modified | "Schema changes" rule now describes the `node-pg-migrate` flow. The legacy `scripts/migrations/YYYY-MM-DD-<slug>.js` convention is documented as "preserved for the lone existing pre-tool migration; not used for new work". |
| `.cursor/rules/db-and-schema.mdc` | modified | § "Schema source of truth" now points at `migrations/` and the `npm run migrate create ...` workflow. The "Until a proper migration tool is adopted" preamble is replaced. |
| `docs/SCHEMA_MAP.md` | modified | Preamble paragraph re-scopes the file: tool-managed schema lives in `migrations/`; this file remains the curated reference for the full prod shape (which still includes the historical `add-*.js` columns the initial backfill doesn't replay). § Regeneration updated to mention `migrations/`. |
| `package-lock.json` | modified | Reflects `node-pg-migrate` + `pg` + transitive deps. |
### File diff highlights
- `migrations/1779853647564_initial-schema.js` is ~155 lines. Seven
`pgm.sql(...)` blocks plus a docstring explaining the
idempotency guarantee, plus a `down()` stub with a long-form
error message.
- `scripts/setup-neon-db.js` net diff: ~+40 / -50. The seven DDL
blocks are deleted; the spawn helper + the `POSTGRES_URL`
guard + the updated message copy are added.
- `package.json` adds one `scripts` line + two `devDependencies`
entries.
- The doc / rule edits are 1-2 paragraphs each.
## Verification plan
1. `npm run lint` → exit 1 with 128 problems (baseline preserved).
The new migration file MUST be lint-clean (no new ignore patterns
in `eslint.config.mjs`).
2. `npm run test:run` → 21/21 pass. Vitest does not touch the
migration surface; the run must stay green.
3. `node --check migrations/1779853647564_initial-schema.js` → exit 0
(syntax-valid).
4. `node --check scripts/setup-neon-db.js` → exit 0.
5. Dynamic import of the migration file: `node -e "import('./migrations/...').then(m => m.down())"`
should throw with the documented error message (proves D7 is wired
correctly).
6. `npm run migrate -- --help` returns the node-pg-migrate help text
through the wrapper (proves the wrapper's flag chain is shell-parseable).
**Live test against a Neon branch: NOT performed in this convoy.** The
parent did not have a throwaway Neon branch available, and the convoy
spec authorizes documenting the gap. Operator's optional post-merge
verification:
```
DATABASE_URL=<neon-branch> npm run migrate up
# expect: applies the initial migration; pgmigrations row appears
DATABASE_URL=<neon-branch> npm run migrate down # expect: throws hard stub
ADMIN_INITIAL_PASSWORD=<...> POSTGRES_URL=<neon-branch> npm run setup-db
# expect: migrate up logs (no-op since the migration is already applied), then admin user seeded
```
(Note the `--database-url-var POSTGRES_URL` wrapper means
`POSTGRES_URL` is the var the operator sets; the `DATABASE_URL`
calls above are illustrative of how a stock node-pg-migrate would be
invoked. For the wrapper, set `POSTGRES_URL` in `.env.local` and just
run `npm run migrate up`.)
## Risks
### R1 — Prod schema drift from `setup-neon-db.js` DDL
The 27 historical `scripts/add-*.js` jobs added columns + tables to
prod that `setup-neon-db.js` never created (`user_settings`,
`user_avatars`, `user_favorites`, `collection_permissions`,
`collection_activity`, `invitations`, plus many `users` and
`collections` columns — see `docs/SCHEMA_MAP.md`). The initial
backfill captures only the post-`setup-neon-db.js` shape.
**Why this is safe:** `CREATE TABLE IF NOT EXISTS` is a no-op on
existing tables. Running `npm run migrate up` against the prod DB
applies the backfill migration (recording it in `pgmigrations`)
without touching the existing schema. Future migrations can ALTER /
CREATE freely from this baseline.
**Why this is not safe for fresh-env onboarding:** a brand-new Neon
branch onboarded via `npm install``npm run setup-db` will end up
with the bootstrap 7-table shape only — none of the historical
add-*.js columns will be present. Most code paths assume those
columns exist (see `docs/SCHEMA_MAP.md` for the full shape).
Mitigation: the queued `reconcile-historical-add-scripts` follow-up
convoy will fold the historical effects into the migration history.
Until then, fresh-env onboarding still requires either a Neon branch
of an existing prod-shaped DB, or a manual replay of the
historically-applied scripts (which is the pre-convoy status quo —
this convoy does not regress that situation).
### R2 — Down-migration is a hard stub on the initial backfill
By design (D7). Documented in the migration file's `down()`
docstring and in this convoy. Future migrations (`alter-foo-add-bar`,
etc.) should have real `down()`s for safe rollback; the stub is
specific to the initial backfill.
### R3 — CI does not exercise migrations
By design (D6). The first signal that a new migration is broken is
the developer's local `npm run migrate up` against their dev branch.
Mitigation: surface as `wire-migrate-into-ci` follow-up convoy.
**Operator-visible consequence:** a PR that adds a syntactically-valid
but logically-broken migration (e.g. `ALTER TABLE non_existent_table`)
will pass CI; the breakage shows up the first time `npm run setup-db`
or `npm run migrate up` is run against an env. Pre-deploy, this is
caught by the developer's own dev-loop. Post-deploy, the migration's
failure on prod is `setup-neon-db.js` exiting non-zero with the
wrapped error message — visible in the Vercel deploy logs.
### R4 — Operator must seed `POSTGRES_URL` for any migrate invocation
`npm run migrate up` requires `POSTGRES_URL` in `.env.local` (or set
in the calling environment). If unset, `node-pg-migrate` errors with
its standard "No database connection string" message; the wrapper
doesn't add a friendlier pre-flight check (matching node-pg-migrate's
default behavior is fine — operators running migrations are by
definition operating on a known DB). `setup-neon-db.js` does add an
explicit `POSTGRES_URL` check before spawning the migration runner,
so the `npm run setup-db` happy path produces a useful error.
### R5 — New `npm audit` vulnerabilities surface via `node-pg-migrate`'s
glob + yargs transitive deps
`node-pg-migrate@8.0.4` pulls in `glob@~11.1.0` + `yargs@~17.7.0`,
which transitively bring in older `brace-expansion`, `minimatch`, and
`picomatch` versions with known advisories. `npm audit` reports 11
vulnerabilities (6 moderate, 5 high) at install time. These are
**all in dev-only paths** (the migration tool runs in scripts/CI,
never in the deployed Next.js bundle) and the affected APIs (glob's
shell-injection CLI; brace-expansion's ReDoS) are not exercised by
node-pg-migrate's call sites. The convoy spec says "do NOT bump
unrelated deps", so the audit fix is deferred. Surface as a
follow-up if a security audit flags this surface specifically.
### R6 — Migration `1779853647564_initial-schema` timestamp is fixed
Once committed and applied anywhere, the file name + the migration's
contents are pinned in the `pgmigrations` table. Mutating either
silently corrupts any env that has the prior version recorded. This
is `node-pg-migrate`'s standard contract; documented inline in the
migration file's docstring. A future schema correction needs a NEW
dated migration, not an edit to this file.
## Operator runbook (new flow)
### Schema change
```bash
# 1. Generate a migration file (JS template, timestamp-prefixed)
npm run migrate create add-foo-column -- -j js
# 2. Edit the generated file under migrations/<timestamp>_add-foo-column.js
# - Put DDL in up() via pgm.sql(`ALTER TABLE ...`)
# - Write a real down() if rollback is safe; otherwise a throwing stub.
# 3. Apply locally against the dev DB (POSTGRES_URL from .env.local)
npm run migrate up
# 4. Update docs/SCHEMA_MAP.md to reflect the schema change.
# 5. Commit the migration file + docs together. CI runs lint + vitest
# only (D6); the migration itself isn't exercised in CI yet.
# 6. After merge + deploy, an operator runs the migration against prod:
# POSTGRES_URL=<prod> npm run migrate up
# (Or wait for the next setup-db invocation; it chains migrate up.)
```
### Onboarding a new env
```bash
# 1. Clone + install
git clone <repo>
cd tcg-vault
npm install
# 2. Seed .env.local (POSTGRES_URL, JWT_SECRET, ADMIN_INITIAL_PASSWORD).
# 3. Run setup-db — chains migrate up, then seeds the admin user.
npm run setup-db
```
### Re-running setup against an existing env
Idempotent on both halves:
- The initial backfill migration uses `CREATE TABLE IF NOT EXISTS`,
so it's a no-op on tables that already exist.
- The admin-row INSERT uses `ON CONFLICT (email) DO NOTHING`.
The operator caveat from `drop-public-setup` Brief 1 still applies:
re-running setup-db does NOT rotate an existing admin row's password.
See README § "First-time admin setup" → operator-rotation note.
### Recovering from a failed migration
If `npm run migrate up` fails partway through (unlikely for the initial
backfill since each CREATE is independent; possible for future
migrations with multi-statement up()s), the operator's options:
1. **Fix forward**: edit the failing migration file, re-run
`npm run migrate up`. node-pg-migrate's default
`--single-transaction true` flag means a failure rolls back the
transaction, so the DB is left in the pre-migration state and the
`pgmigrations` row is NOT recorded. Re-running picks up cleanly.
2. **Skip a broken migration** (last resort): `npm run migrate up -- --fake`
marks pending migrations as applied without running them. Use this
ONLY if the schema state is already correct outside the tool's view
(e.g., the migration was applied manually via psql). Document any
`--fake` use in the convoy/PR that caused it.
## Follow-ups
- **`wire-migrate-into-ci`** (priority: P2 CI infra). Add a CI job that
runs `npm run migrate up` against a test DB (either a dedicated
Neon branch + secret, or a Postgres service container). Catches
syntactically-invalid migrations + most logical errors at PR time.
Deferred per D6.
- **`reconcile-historical-add-scripts`** (priority: P1 quality —
needed for fresh-env onboarding). Fold the effects of the 27
historical `scripts/add-*.js` / `scripts/fix-*.js` / `scripts/seed-*.js`
jobs into the migration history so a brand-new Neon branch can be
onboarded by `npm install``npm run setup-db` alone (without
manually replaying the historical scripts). Multi-PR: one
migration per logical change, ideally generated by reading the
scripts' SQL and re-shaping into idempotent `pgm.sql(...)` blocks
(with `IF NOT EXISTS` / `IF EXISTS` guards so re-application is
safe).
- **`retire-graveyard-scripts-after-audit`** (priority: P3 polish,
blocked on `reconcile-historical-add-scripts`). Once the migration
history captures all historical effects, the legacy `scripts/add-*.js`
/ `scripts/fix-*.js` / `scripts/seed-*.js` files can be deleted (or
moved to `scripts/historical/`). They remain no-go-zones until that
cleanup convoy lands.
- **`audit-node-pg-migrate-transitive-deps`** (priority: P3 hygiene).
`npm audit` reports 11 vulnerabilities (6 moderate, 5 high) coming
from node-pg-migrate's glob + yargs transitive deps. All in dev-only
paths; not exercised by node-pg-migrate's call sites. Surface only
if a security audit specifically flags this surface, or if
node-pg-migrate ships a v9 that updates the transitive tree.
- **`add-migration-template`** (priority: P3 DX). Add a custom template
via `--template-file-name` so generated migrations include the
project's preferred docstring shape + a reminder about
`docs/SCHEMA_MAP.md` updates. Surface if migration authoring proves
inconsistent.
## As-shipped
Single squash commit `de9f334` (PR #32, merged 2026-05-27T04:01:59Z
UTC / local 2026-05-26). Parent-owned end-to-end per the § Subagent / multitask
footnote — no architect or implementer subagent dispatched. The
convoy spec pre-ratified each Decision's recommended path, and the
implementation surface was a small set of well-bounded file edits
following the spec's "The change (implementation shape)" checklist
verbatim. **AGENTS.md Gotcha #6 flipped from open → RESOLVED in this
same wave's post-convoy doc-writer pass.**
**Diff: 10 files, +1230 / -136.** The 1230-addition figure includes
`.convoys/migration-tool.md` (the planning document, ~600 lines,
committed atomically), `migrations/1779853647564_initial-schema.js`
(~155 lines for the backfill migration), the doc / rule / skill
edits, and `package-lock.json` churn for the `node-pg-migrate@^8.0.4`
+ `pg@^8.21.0` install (plus transitive deps).
### Decisions ratified at gate 1
All seven decisions landed verbatim from the convoy spec's
recommendations. No mid-execution surprises that would have routed
back through an architect bounce.
### As-shipped surface
- **`migrations/` directory created at repo root** (was: empty / nonexistent).
- **`migrations/1779853647564_initial-schema.js`** added. ~155 lines.
Seven `pgm.sql(\`CREATE TABLE IF NOT EXISTS ...\`)` blocks
reproducing `scripts/setup-neon-db.js`'s 7-table DDL verbatim.
Down-migration is a hard stub that throws with a long-form error
message naming the alternative (Neon branch + forward-apply).
Lint-clean (no new ignore patterns in `eslint.config.mjs`).
- **`scripts/setup-neon-db.js`** refactored. The seven `await sql\`CREATE TABLE IF NOT EXISTS\`` blocks are removed (DDL now lives in the migration). A `runMigrations()` helper is added that spawns `npm run migrate up` via `node:child_process.spawn({ stdio: 'inherit', shell: false })` and rejects with a wrapped error on non-zero exit. A `POSTGRES_URL` pre-flight check is added (was previously implicit). The success/error message copy is updated to mention the migration step explicitly so the operator's mental model matches the new pipeline. The admin-row INSERT (with `ON CONFLICT (email) DO NOTHING`) and the `ADMIN_INITIAL_PASSWORD` env-var check are preserved verbatim.
- **`package.json`**: `scripts.migrate` added (`node-pg-migrate --database-url-var POSTGRES_URL --envPath .env.local --migrations-dir migrations --verbose`). `devDependencies` adds `node-pg-migrate@^8.0.4` + `pg@^8.21.0`.
- **`README.md`** updated: § Installation step 4 documents the migration chain; new § "Schema changes (post-`migration-tool` convoy)" explains the create/edit/up/commit flow; § "First-time admin setup" mentions the migrate step.
- **`AGENTS.md`** updated: § 3 Conventions gains a "Schema changes" bullet pointing at the new flow + `.convoys/migration-tool.md`; § 4 Gotcha #6 flipped from open → RESOLVED with the as-shipped paragraph (tool / dir / tracking-table / idempotency notes).
- **`.cursor/rules/no-go-zones.mdc`** updated: "Schema changes" rule rewritten to describe the `node-pg-migrate` flow; legacy `scripts/migrations/YYYY-MM-DD-<slug>.js` convention documented as "preserved for the lone existing pre-tool migration; not used for new work".
- **`.cursor/rules/db-and-schema.mdc`** updated: § "Schema source of truth" rewritten to point at `migrations/` and the `npm run migrate create ...` workflow.
- **`docs/SCHEMA_MAP.md`** updated: preamble re-scopes the file (tool-managed schema lives in `migrations/`; this file remains the curated reference for the full prod shape including historical add-*.js columns); § Regeneration updated.
### Verification (all gates green pre-PR)
- `npm run lint` → exit 1 with **128 problems** (baseline preserved, zero regression). The new migration file is lint-clean; no new ignore patterns added.
- `npm run test:run`**21/21 pass** in ~1.3s (4 test files, no rate-limit / migration touchpoint).
- `node --check migrations/1779853647564_initial-schema.js` → exit 0.
- `node --check scripts/setup-neon-db.js` → exit 0.
- Module-load + `down()` throw verification:
`node -e "import('./migrations/1779853647564_initial-schema.js').then(m => m.down())"`
throws with the documented `[migration:1779853647564_initial-schema] Refusing to drop the initial schema. ...` message.
- `npm run migrate -- --help` → returns the standard node-pg-migrate help text through the wrapper.
- CI on PR #32: Lint ✓ (128 problems at convoy time; the new lint baseline post-PR-#31 is 125 — see § What did NOT change below for the version note) | Vitest 21/21 ✓ | Playwright smoke 3/3 ✓ | `forbidden-endpoints` ✓ | `forbidden-cors-headers` ✓ | Vercel preview deploy ✓ | Aggregate gate ✓
- `Screenshot diff`: not triggered (PR #32's diff is `migrations/**` + `scripts/**` + `package.json` + docs / rules / `package-lock.json` — none of which matches the visual-diff `paths:` filter; the post-PR-#26 `!pages/api/**` exclusion is not even relevant here).
**Spec deviation:** none. All seven decisions landed verbatim from
the convoy spec's recommendations at gate 1. The verification gates
match § Verification plan exactly.
### Live verification status
**Deferred** per the convoy spec. The parent did not have a throwaway
Neon branch available. The operator's optional post-merge verification
sequence is documented in § Operator runbook → "Re-running setup
against an existing env".
### Operator action required going forward
**None for the convoy itself.** The migration is idempotent against
the existing prod schema (CREATE TABLE IF NOT EXISTS no-ops on
existing tables; the `pgmigrations` row is the only DB-side change).
No new env vars; no new secrets. The existing `POSTGRES_URL` +
`ADMIN_INITIAL_PASSWORD` contract is preserved.
**Optional but recommended post-merge:**
1. The next deploy that runs `setup-neon-db.js` will silently apply
the backfill migration (recording it in `pgmigrations`). No
operator action; this is just-in-time chained.
2. To pre-apply the migration without re-running the seed step:
`POSTGRES_URL=<...> npm run migrate up`.
### Cross-validation finding
Same shape as the `add-rate-limiting` cross-validation finding: the
existing `Playwright smoke` 3/3 spec defends the post-migration-tool
deployment without anyone writing a dedicated test. Smoke calls
`/api/health` once per run; `/api/health` reads from the `users` table
(via `lib/database.js` health-check query — verify with the smoke run
on the PR) which is created by both the initial backfill AND the
pre-existing `scripts/setup-neon-db.js` DDL, so smoke's signal is
invariant to which mechanism populated the schema. Sixth consecutive
convoy where the same 3-test smoke spec defends the auth surface
through a sweeping change (PR #15 Layout default-user → PR #19
CORS-tighten → PR #20 rate-limiting → PR #21 pick-a-name → PR #25
reset-db-fix → this convoy).
### What did NOT change
- `lib/database.js`, `lib/permission-middleware.js`, `lib/auth-secret.js`, `lib/rate-limit.js` — no auth or runtime surface touched.
- `pages/api/**/*.js` — no API handlers touched.
- `test/**` — no test surface touched (Decision-6-equivalent: per-route handler tests for the migration runner would be valuable but are out of scope; the migration's correctness is verified via the operator runbook's manual sequence).
- `scripts/migrations/2026-05-24-rename-admin-email.js` — preserved verbatim; not migrated into the new `migrations/` directory because it's already-applied history (the no-go-zones rule covers it).
- `.github/workflows/**` — no new CI job (Decision 6 defers `wire-migrate-into-ci` to a follow-up convoy).
- `package-lock.json` semantics — only adds the new transitive deps; no version bumps to existing deps.
- The legacy 27 `scripts/add-*.js` / `scripts/fix-*.js` / `scripts/seed-*.js` files — all preserved as append-only history per the no-go-zones rule.
### Subagent / multitask footnote
This convoy ran with the parent owning architecture + implementation
end-to-end (no architect or implementer subagent dispatch). The
convoy spec pre-ratified each Decision's recommended path, and the
implementation surface was a small set of well-bounded file edits
following the spec's "The change (implementation shape)" checklist
verbatim. A mid-execution worktree migration was required (the parent
opened the convoy from the main worktree, which was incidentally
checked out on a sibling branch `convoy/purge-weak-creds-from-helpers`
due to a parallel agent's activity; the parent stashed its work,
created `tcg-vault-worktrees/migration-tool` as a fresh worktree on
the canonical `convoy/migration-tool` branch, re-applied the work
there via a tracked-changes patch + an untracked-files tar, and
continued cleanly). No file content was lost in that migration; the
git history is single-branch from the worktree's perspective.