593 lines
31 KiB
Markdown
593 lines
31 KiB
Markdown
|
|
---
|
||
|
|
name: migration-tool
|
||
|
|
classification: quality
|
||
|
|
priority: P1 (launch sequence step 7)
|
||
|
|
success_metric: |
|
||
|
|
Schema changes ship as a migration file under `migrations/` at the
|
||
|
|
repo root and are applied by `npm run migrate up`. The initial
|
||
|
|
backfill migration reproduces `scripts/setup-neon-db.js`'s 7-table
|
||
|
|
bootstrap DDL verbatim. `npm run setup-db` invokes the migration
|
||
|
|
runner before seeding the admin user. `.convoys/ship-readiness.md`
|
||
|
|
P1 #11 ("No migration tool — `scripts/add-*.js` graveyard") flips
|
||
|
|
from open → RESOLVED.
|
||
|
|
skip:
|
||
|
|
- role-design-system-auditor
|
||
|
|
- role-a11y-auditor
|
||
|
|
- role-ux-reviewer
|
||
|
|
- role-ia-architect
|
||
|
|
status: shipped
|
||
|
|
created: 2026-05-26
|
||
|
|
shipped: 2026-05-26
|
||
|
|
parent: ship-readiness
|
||
|
|
addresses: P1 #11 (launch sequence step 7)
|
||
|
|
depends_on:
|
||
|
|
- drop-public-setup (setup-neon-db.js was previously CJS — already ESM post-Brief 2; this convoy assumes ESM)
|
||
|
|
- fix-reset-db-script (precedent for the ESM/env-var/seed-line shape that setup-neon-db.js now shares)
|
||
|
|
---
|
||
|
|
|
||
|
|
# Convoy: migration-tool
|
||
|
|
|
||
|
|
Adopt `node-pg-migrate` as the schema-change tool. Backfill a single
|
||
|
|
initial migration that reproduces `scripts/setup-neon-db.js`'s
|
||
|
|
documented bootstrap DDL. Modify `setup-neon-db.js` to invoke the
|
||
|
|
migration runner before seeding the admin user. Document the new
|
||
|
|
convention.
|
||
|
|
|
||
|
|
## Background — the graveyard problem
|
||
|
|
|
||
|
|
`AGENTS.md` Gotcha #6 (pre-convoy) flagged 27+ scripts in `scripts/`
|
||
|
|
of the form `add-*.js` / `fix-*.js` / `seed-*.js`, each a one-shot
|
||
|
|
DDL change applied once to prod with no idempotency tracking and no
|
||
|
|
rollback. Onboarding a fresh env required re-running every script in
|
||
|
|
the right order; there was no way to know what had been run on a
|
||
|
|
given Neon branch; every new column was at risk of being missed in
|
||
|
|
prod.
|
||
|
|
|
||
|
|
The lone existing "tool-shaped" migration was
|
||
|
|
`scripts/migrations/2026-05-24-rename-admin-email.js` (`pick-a-name`
|
||
|
|
convoy). It uses ESM, is idempotent, and is UNIQUE-collision-safe —
|
||
|
|
the right shape for a single migration script, but it is NOT wrapped
|
||
|
|
by any migration tool, so re-running it is the operator's
|
||
|
|
responsibility and the operator gets no signal about whether the
|
||
|
|
migration has already been applied to a given env.
|
||
|
|
|
||
|
|
The `.cursor/rules/no-go-zones.mdc` rule (pre-convoy) documented the
|
||
|
|
ad-hoc `scripts/migrations/YYYY-MM-DD-<slug>.js` convention as a
|
||
|
|
placeholder *"until a real migration tool is adopted"*. This convoy
|
||
|
|
adopts that tool.
|
||
|
|
|
||
|
|
## Decisions
|
||
|
|
|
||
|
|
Seven architect decisions. D1-D2 + D4-D5 + D7 are parent-self-ratified
|
||
|
|
following the convoy spec's recommended choices verbatim (the
|
||
|
|
recommendations included tuning evidence and matched the project's
|
||
|
|
existing patterns). D3 + D6 are mechanically determined by D1-D2.
|
||
|
|
This is a single-implementer convoy; no operator gate was needed
|
||
|
|
because the spec pre-ratified each Decision's recommended path.
|
||
|
|
|
||
|
|
### D1 — Tool: `node-pg-migrate@^8`
|
||
|
|
|
||
|
|
**Ratified: `node-pg-migrate@^8.0.4`** (the version that resolves at
|
||
|
|
install time; pinned to `^8` in `devDependencies`).
|
||
|
|
|
||
|
|
Considered alternatives:
|
||
|
|
|
||
|
|
| Tool | Why rejected |
|
||
|
|
| --- | --- |
|
||
|
|
| `drizzle-kit` | Schema-as-code in TypeScript; would force broader TS adoption than `AGENTS.md` Gotcha #9 allows (TS is a devDep only for the `eslint-config-next` chain). Out of scope. |
|
||
|
|
| `prisma migrate` | Whole-ORM adoption is a multi-month surface change. Out of scope. |
|
||
|
|
| `kysely` migrations | TypeScript-first; same TS-adoption issue as drizzle-kit. Out of scope. |
|
||
|
|
| Hand-rolled in-house tool | Reinvents the tracking-table + locking + dry-run + idempotency-mark surface; pure cost, zero novel value. |
|
||
|
|
|
||
|
|
`node-pg-migrate` matches the repo's existing style:
|
||
|
|
|
||
|
|
- JavaScript-only (no `tsconfig.json` required; ESM migrations work
|
||
|
|
out of the box because `package.json` has `"type": "module"`).
|
||
|
|
- Raw SQL-friendly via `pgm.sql(...)` — no schema-as-code DSL to
|
||
|
|
learn. The pre-existing `2026-05-24-rename-admin-email.js`
|
||
|
|
migration is hand-written SQL-shaped JS; the new tool wraps that
|
||
|
|
exact shape.
|
||
|
|
- Lightweight: ~533 KB unpacked; only `glob@~11.1.0` + `yargs@~17.7.0`
|
||
|
|
as direct deps.
|
||
|
|
- Lowest-magic option in the candidate set: no codegen, no
|
||
|
|
declarative diff, no opinions about file layout beyond a
|
||
|
|
conventional `migrations/` folder.
|
||
|
|
|
||
|
|
Cost: brings in `pg@^8.21.0` as a peer dep (needed because
|
||
|
|
`node-pg-migrate` uses the standard TCP `pg` client, not Neon's HTTP
|
||
|
|
driver). That's an extra ~3 MB of `node_modules`; no runtime cost
|
||
|
|
because the migration tool is dev-only / scripts-only and never
|
||
|
|
loads in the Next.js bundle.
|
||
|
|
|
||
|
|
### D2 — Migrations directory: `migrations/` at repo root
|
||
|
|
|
||
|
|
**Ratified: `migrations/` at the repo root.**
|
||
|
|
|
||
|
|
`scripts/migrations/` already exists as the historical placeholder
|
||
|
|
folder (housing `2026-05-24-rename-admin-email.js`). Per the convoy
|
||
|
|
spec's suggested split, the new tool-wrapped migrations live in a
|
||
|
|
fresh top-level `migrations/` directory:
|
||
|
|
|
||
|
|
- Separates the tool-wrapped artifacts from the historical
|
||
|
|
placeholder. The placeholder folder is preserved for the audit
|
||
|
|
trail of the lone pre-tool migration; agents reading
|
||
|
|
`scripts/migrations/` see "this is history, NOT how new schema
|
||
|
|
changes ship".
|
||
|
|
- Matches `node-pg-migrate`'s default `--migrations-dir migrations`
|
||
|
|
(zero CLI noise for the common case).
|
||
|
|
- Keeps tool-managed state out of `scripts/`, which already houses 27
|
||
|
|
append-only legacy jobs that are flagged as no-go-zones.
|
||
|
|
|
||
|
|
### D3 — Tracking table: default `pgmigrations`
|
||
|
|
|
||
|
|
**Ratified: accept the default.** No existing table in the schema
|
||
|
|
collides with `pgmigrations` (verified by reading
|
||
|
|
`scripts/setup-neon-db.js`'s 7-table DDL and by cross-referencing
|
||
|
|
`docs/SCHEMA_MAP.md`). Zero CLI flag, zero documentation overhead.
|
||
|
|
|
||
|
|
### D4 — Backfill strategy: hand-translate `setup-neon-db.js`
|
||
|
|
|
||
|
|
**Ratified: hand-translate `scripts/setup-neon-db.js`'s DDL into the
|
||
|
|
initial backfill migration.**
|
||
|
|
|
||
|
|
Considered alternative: run `pg_dump --schema-only` against a fresh
|
||
|
|
`npm run setup-db` to capture the bootstrap shape mechanically. The
|
||
|
|
mechanical approach would be marginally more "guaranteed correct"
|
||
|
|
versus the hand-translated copy, but requires standing up a
|
||
|
|
throwaway Neon branch, dotenv-wiring `pg_dump`, sanitizing the
|
||
|
|
output (stripping owner/permissions lines that `pg_dump` adds), and
|
||
|
|
manually accepting the resulting normalized SQL into the migration
|
||
|
|
file. That's more moving parts than a verbatim copy from a single
|
||
|
|
documented file.
|
||
|
|
|
||
|
|
The hand-translation is mechanically simple: each `await sql\`\`` block
|
||
|
|
in `setup-neon-db.js` becomes one `pgm.sql(\`...\`)` call in the
|
||
|
|
migration's `up()`. The seven `CREATE TABLE IF NOT EXISTS` blocks
|
||
|
|
are preserved verbatim (including column order, types, defaults, FK
|
||
|
|
clauses, and the UNIQUE constraints). The admin-row INSERT in
|
||
|
|
`setup-neon-db.js` is NOT replicated into the migration — that's
|
||
|
|
the seed step, which stays in `setup-neon-db.js` (per D5).
|
||
|
|
|
||
|
|
**Documented assumption:** `setup-neon-db.js` is the bootstrap
|
||
|
|
source-of-truth that the project has shipped with since first commit.
|
||
|
|
If the prod schema has drifted from that bootstrap shape (which it
|
||
|
|
HAS — the 27 historical `scripts/add-*.js` jobs added many columns
|
||
|
|
that the bootstrap script never created), the drift is out-of-scope
|
||
|
|
for this convoy. The backfill migration captures the
|
||
|
|
`setup-neon-db.js`-shape only; reconciling the full prod shape into
|
||
|
|
the migration history is the queued
|
||
|
|
`reconcile-historical-add-scripts` follow-up convoy.
|
||
|
|
|
||
|
|
The choice is documented as a known limitation in `docs/SCHEMA_MAP.md`'s
|
||
|
|
new preamble: *"the initial backfill captures only the
|
||
|
|
post-`setup-neon-db.js` shape ... if a fresh env needs the full
|
||
|
|
historical column set, a follow-up convoy ... will fold the historical
|
||
|
|
effects into the migration history; until then this file remains the
|
||
|
|
curated reference for the full prod shape."*
|
||
|
|
|
||
|
|
The use of `CREATE TABLE IF NOT EXISTS` in the backfill (matching
|
||
|
|
`setup-neon-db.js` verbatim) means the initial migration is safe to
|
||
|
|
run against existing envs — every CREATE is a no-op on tables that
|
||
|
|
already exist (the historical add-*.js columns are preserved). Only
|
||
|
|
the `pgmigrations` row changes.
|
||
|
|
|
||
|
|
### D5 — Bootstrap reconciliation: split DDL (migrations) from seed (`setup-neon-db.js`)
|
||
|
|
|
||
|
|
**Ratified: split.** `setup-neon-db.js` now:
|
||
|
|
|
||
|
|
1. Validates `ADMIN_INITIAL_PASSWORD` is set (fail loud BEFORE touching the DB).
|
||
|
|
2. Validates `POSTGRES_URL` is set (new — was previously implicit).
|
||
|
|
3. Spawns `npm run migrate up` via `node:child_process.spawn` with
|
||
|
|
`stdio: 'inherit'`. If migrate exits non-zero, setup-db exits non-zero
|
||
|
|
with a wrapping error message that surfaces the failing exit code +
|
||
|
|
signal and points the operator at `npm run migrate up` to re-try just
|
||
|
|
the migration step.
|
||
|
|
4. Connects to Neon (via the existing `@neondatabase/serverless` HTTP driver)
|
||
|
|
and runs the admin-row INSERT with `ON CONFLICT (email) DO NOTHING`.
|
||
|
|
|
||
|
|
The seven `await sql\`CREATE TABLE IF NOT EXISTS ...\`` blocks are
|
||
|
|
removed from `setup-neon-db.js` — they now live in
|
||
|
|
`migrations/1779853647564_initial-schema.js`. The script's success/error
|
||
|
|
message copy is updated to mention the migration step explicitly so
|
||
|
|
the operator's mental model matches the new pipeline.
|
||
|
|
|
||
|
|
Rejected alternative: keep all DDL in `setup-neon-db.js`, have the
|
||
|
|
migration runner be a separate `npm run migrate` invocation operators
|
||
|
|
remember to call. This would leave the DDL-ownership ambiguity intact
|
||
|
|
(two sources of truth) and require operators to know to call both
|
||
|
|
scripts in the right order — exactly the kind of operator-burden that
|
||
|
|
the convoy is trying to remove.
|
||
|
|
|
||
|
|
### D6 — CI integration: defer
|
||
|
|
|
||
|
|
**Ratified: defer to `wire-migrate-into-ci` follow-up convoy.**
|
||
|
|
|
||
|
|
Adding a CI job that runs `npm run migrate up` against a test Neon
|
||
|
|
branch (or against a temporary Postgres container) requires either:
|
||
|
|
|
||
|
|
- A dedicated test Neon branch + a `MIGRATE_TEST_DATABASE_URL` secret
|
||
|
|
in GitHub Actions, plus branch-reset logic so successive PRs don't
|
||
|
|
see each other's migrations.
|
||
|
|
- A Postgres service container in the workflow YAML, which means a
|
||
|
|
~30s container-start tax on every PR and a CI-specific code path for
|
||
|
|
the migration runner.
|
||
|
|
|
||
|
|
Both are real work. The convoy spec explicitly authorizes deferring
|
||
|
|
this to a follow-up — surfaced as `wire-migrate-into-ci` in the
|
||
|
|
follow-ups list below. **Risk acknowledged:** until that follow-up
|
||
|
|
lands, new migration files are validated at code-review time only
|
||
|
|
(via `node --check`, the implementer's local `npm run migrate up`
|
||
|
|
against their dev branch, and any operator-led pre-deploy run).
|
||
|
|
|
||
|
|
### D7 — Down-migration shape on the initial backfill: hard stub
|
||
|
|
|
||
|
|
**Ratified: hard stub that throws.** Per the convoy spec's recommended
|
||
|
|
"too risky to drop prod schema" path.
|
||
|
|
|
||
|
|
Rolling back the initial schema would drop every users / cards /
|
||
|
|
collections / decks row in the database — including the
|
||
|
|
`pgmigrations` row itself. There is no realistic ops scenario where
|
||
|
|
this is the right thing to do; if a developer genuinely needs a
|
||
|
|
clean schema for testing, branching the Neon database is instant +
|
||
|
|
cheap and produces a strictly better outcome (zero data loss for
|
||
|
|
other developers using the same branch).
|
||
|
|
|
||
|
|
The stub's error message names the migration explicitly, explains the
|
||
|
|
risk, and points at the recommended alternative (Neon branch +
|
||
|
|
forward-apply). Future migrations that touch one of the seven
|
||
|
|
bootstrap tables should write their OWN dated migration with a real
|
||
|
|
`down()` — they do NOT need to re-enable the down on this initial
|
||
|
|
backfill.
|
||
|
|
|
||
|
|
## The fix
|
||
|
|
|
||
|
|
| File | Action | Purpose |
|
||
|
|
| --- | --- | --- |
|
||
|
|
| `migrations/1779853647564_initial-schema.js` | new | The backfill migration. `up()` runs seven `pgm.sql(\`CREATE TABLE IF NOT EXISTS ...\`)` blocks reproducing `scripts/setup-neon-db.js`'s 7-table DDL verbatim (users / cards / user_cards / collections / collection_cards / decks / deck_cards). `down()` throws (D7). `shorthands` is `undefined`. |
|
||
|
|
| `package.json` | modified | Adds `"migrate": "node-pg-migrate --database-url-var POSTGRES_URL --envPath .env.local --migrations-dir migrations --verbose"` to `scripts`. Adds `node-pg-migrate@^8.0.4` and `pg@^8.21.0` to `devDependencies`. `setup-db` script unchanged at the YAML layer; its behavior changes per D5. |
|
||
|
|
| `scripts/setup-neon-db.js` | modified | Splits DDL (now in the migration) from seed (admin-row INSERT). Adds a `POSTGRES_URL` env-var check (was previously implicit). Adds a `runMigrations()` helper that spawns `npm run migrate up` and rejects with a wrapped error on non-zero exit. Updates the success/error message copy to mention the migration step. |
|
||
|
|
| `README.md` | modified | § Installation step 4 now mentions that `setup-db` chains the migration runner. New § "Schema changes" documents the `npm run migrate create ...` → edit → `npm run migrate up` → commit flow. § "First-time admin setup" mentions the migrate step. |
|
||
|
|
| `AGENTS.md` | modified | § 3 Conventions: new "Schema changes (post-`migration-tool`)" bullet pointing at the new flow. § 4 Gotchas: Gotcha #6 flipped from open → RESOLVED with the as-shipped paragraph (tool / dir / tracking-table / idempotency notes). |
|
||
|
|
| `.cursor/rules/no-go-zones.mdc` | modified | "Schema changes" rule now describes the `node-pg-migrate` flow. The legacy `scripts/migrations/YYYY-MM-DD-<slug>.js` convention is documented as "preserved for the lone existing pre-tool migration; not used for new work". |
|
||
|
|
| `.cursor/rules/db-and-schema.mdc` | modified | § "Schema source of truth" now points at `migrations/` and the `npm run migrate create ...` workflow. The "Until a proper migration tool is adopted" preamble is replaced. |
|
||
|
|
| `docs/SCHEMA_MAP.md` | modified | Preamble paragraph re-scopes the file: tool-managed schema lives in `migrations/`; this file remains the curated reference for the full prod shape (which still includes the historical `add-*.js` columns the initial backfill doesn't replay). § Regeneration updated to mention `migrations/`. |
|
||
|
|
| `package-lock.json` | modified | Reflects `node-pg-migrate` + `pg` + transitive deps. |
|
||
|
|
|
||
|
|
### File diff highlights
|
||
|
|
|
||
|
|
- `migrations/1779853647564_initial-schema.js` is ~155 lines. Seven
|
||
|
|
`pgm.sql(...)` blocks plus a docstring explaining the
|
||
|
|
idempotency guarantee, plus a `down()` stub with a long-form
|
||
|
|
error message.
|
||
|
|
- `scripts/setup-neon-db.js` net diff: ~+40 / -50. The seven DDL
|
||
|
|
blocks are deleted; the spawn helper + the `POSTGRES_URL`
|
||
|
|
guard + the updated message copy are added.
|
||
|
|
- `package.json` adds one `scripts` line + two `devDependencies`
|
||
|
|
entries.
|
||
|
|
- The doc / rule edits are 1-2 paragraphs each.
|
||
|
|
|
||
|
|
## Verification plan
|
||
|
|
|
||
|
|
1. `npm run lint` → exit 1 with 128 problems (baseline preserved).
|
||
|
|
The new migration file MUST be lint-clean (no new ignore patterns
|
||
|
|
in `eslint.config.mjs`).
|
||
|
|
2. `npm run test:run` → 21/21 pass. Vitest does not touch the
|
||
|
|
migration surface; the run must stay green.
|
||
|
|
3. `node --check migrations/1779853647564_initial-schema.js` → exit 0
|
||
|
|
(syntax-valid).
|
||
|
|
4. `node --check scripts/setup-neon-db.js` → exit 0.
|
||
|
|
5. Dynamic import of the migration file: `node -e "import('./migrations/...').then(m => m.down())"`
|
||
|
|
should throw with the documented error message (proves D7 is wired
|
||
|
|
correctly).
|
||
|
|
6. `npm run migrate -- --help` returns the node-pg-migrate help text
|
||
|
|
through the wrapper (proves the wrapper's flag chain is shell-parseable).
|
||
|
|
|
||
|
|
**Live test against a Neon branch: NOT performed in this convoy.** The
|
||
|
|
parent did not have a throwaway Neon branch available, and the convoy
|
||
|
|
spec authorizes documenting the gap. Operator's optional post-merge
|
||
|
|
verification:
|
||
|
|
|
||
|
|
```
|
||
|
|
DATABASE_URL=<neon-branch> npm run migrate up
|
||
|
|
# expect: applies the initial migration; pgmigrations row appears
|
||
|
|
DATABASE_URL=<neon-branch> npm run migrate down # expect: throws hard stub
|
||
|
|
ADMIN_INITIAL_PASSWORD=<...> POSTGRES_URL=<neon-branch> npm run setup-db
|
||
|
|
# expect: migrate up logs (no-op since the migration is already applied), then admin user seeded
|
||
|
|
```
|
||
|
|
|
||
|
|
(Note the `--database-url-var POSTGRES_URL` wrapper means
|
||
|
|
`POSTGRES_URL` is the var the operator sets; the `DATABASE_URL`
|
||
|
|
calls above are illustrative of how a stock node-pg-migrate would be
|
||
|
|
invoked. For the wrapper, set `POSTGRES_URL` in `.env.local` and just
|
||
|
|
run `npm run migrate up`.)
|
||
|
|
|
||
|
|
## Risks
|
||
|
|
|
||
|
|
### R1 — Prod schema drift from `setup-neon-db.js` DDL
|
||
|
|
|
||
|
|
The 27 historical `scripts/add-*.js` jobs added columns + tables to
|
||
|
|
prod that `setup-neon-db.js` never created (`user_settings`,
|
||
|
|
`user_avatars`, `user_favorites`, `collection_permissions`,
|
||
|
|
`collection_activity`, `invitations`, plus many `users` and
|
||
|
|
`collections` columns — see `docs/SCHEMA_MAP.md`). The initial
|
||
|
|
backfill captures only the post-`setup-neon-db.js` shape.
|
||
|
|
|
||
|
|
**Why this is safe:** `CREATE TABLE IF NOT EXISTS` is a no-op on
|
||
|
|
existing tables. Running `npm run migrate up` against the prod DB
|
||
|
|
applies the backfill migration (recording it in `pgmigrations`)
|
||
|
|
without touching the existing schema. Future migrations can ALTER /
|
||
|
|
CREATE freely from this baseline.
|
||
|
|
|
||
|
|
**Why this is not safe for fresh-env onboarding:** a brand-new Neon
|
||
|
|
branch onboarded via `npm install` → `npm run setup-db` will end up
|
||
|
|
with the bootstrap 7-table shape only — none of the historical
|
||
|
|
add-*.js columns will be present. Most code paths assume those
|
||
|
|
columns exist (see `docs/SCHEMA_MAP.md` for the full shape).
|
||
|
|
Mitigation: the queued `reconcile-historical-add-scripts` follow-up
|
||
|
|
convoy will fold the historical effects into the migration history.
|
||
|
|
Until then, fresh-env onboarding still requires either a Neon branch
|
||
|
|
of an existing prod-shaped DB, or a manual replay of the
|
||
|
|
historically-applied scripts (which is the pre-convoy status quo —
|
||
|
|
this convoy does not regress that situation).
|
||
|
|
|
||
|
|
### R2 — Down-migration is a hard stub on the initial backfill
|
||
|
|
|
||
|
|
By design (D7). Documented in the migration file's `down()`
|
||
|
|
docstring and in this convoy. Future migrations (`alter-foo-add-bar`,
|
||
|
|
etc.) should have real `down()`s for safe rollback; the stub is
|
||
|
|
specific to the initial backfill.
|
||
|
|
|
||
|
|
### R3 — CI does not exercise migrations
|
||
|
|
|
||
|
|
By design (D6). The first signal that a new migration is broken is
|
||
|
|
the developer's local `npm run migrate up` against their dev branch.
|
||
|
|
Mitigation: surface as `wire-migrate-into-ci` follow-up convoy.
|
||
|
|
**Operator-visible consequence:** a PR that adds a syntactically-valid
|
||
|
|
but logically-broken migration (e.g. `ALTER TABLE non_existent_table`)
|
||
|
|
will pass CI; the breakage shows up the first time `npm run setup-db`
|
||
|
|
or `npm run migrate up` is run against an env. Pre-deploy, this is
|
||
|
|
caught by the developer's own dev-loop. Post-deploy, the migration's
|
||
|
|
failure on prod is `setup-neon-db.js` exiting non-zero with the
|
||
|
|
wrapped error message — visible in the Vercel deploy logs.
|
||
|
|
|
||
|
|
### R4 — Operator must seed `POSTGRES_URL` for any migrate invocation
|
||
|
|
|
||
|
|
`npm run migrate up` requires `POSTGRES_URL` in `.env.local` (or set
|
||
|
|
in the calling environment). If unset, `node-pg-migrate` errors with
|
||
|
|
its standard "No database connection string" message; the wrapper
|
||
|
|
doesn't add a friendlier pre-flight check (matching node-pg-migrate's
|
||
|
|
default behavior is fine — operators running migrations are by
|
||
|
|
definition operating on a known DB). `setup-neon-db.js` does add an
|
||
|
|
explicit `POSTGRES_URL` check before spawning the migration runner,
|
||
|
|
so the `npm run setup-db` happy path produces a useful error.
|
||
|
|
|
||
|
|
### R5 — New `npm audit` vulnerabilities surface via `node-pg-migrate`'s
|
||
|
|
glob + yargs transitive deps
|
||
|
|
|
||
|
|
`node-pg-migrate@8.0.4` pulls in `glob@~11.1.0` + `yargs@~17.7.0`,
|
||
|
|
which transitively bring in older `brace-expansion`, `minimatch`, and
|
||
|
|
`picomatch` versions with known advisories. `npm audit` reports 11
|
||
|
|
vulnerabilities (6 moderate, 5 high) at install time. These are
|
||
|
|
**all in dev-only paths** (the migration tool runs in scripts/CI,
|
||
|
|
never in the deployed Next.js bundle) and the affected APIs (glob's
|
||
|
|
shell-injection CLI; brace-expansion's ReDoS) are not exercised by
|
||
|
|
node-pg-migrate's call sites. The convoy spec says "do NOT bump
|
||
|
|
unrelated deps", so the audit fix is deferred. Surface as a
|
||
|
|
follow-up if a security audit flags this surface specifically.
|
||
|
|
|
||
|
|
### R6 — Migration `1779853647564_initial-schema` timestamp is fixed
|
||
|
|
|
||
|
|
Once committed and applied anywhere, the file name + the migration's
|
||
|
|
contents are pinned in the `pgmigrations` table. Mutating either
|
||
|
|
silently corrupts any env that has the prior version recorded. This
|
||
|
|
is `node-pg-migrate`'s standard contract; documented inline in the
|
||
|
|
migration file's docstring. A future schema correction needs a NEW
|
||
|
|
dated migration, not an edit to this file.
|
||
|
|
|
||
|
|
## Operator runbook (new flow)
|
||
|
|
|
||
|
|
### Schema change
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 1. Generate a migration file (JS template, timestamp-prefixed)
|
||
|
|
npm run migrate create add-foo-column -- -j js
|
||
|
|
|
||
|
|
# 2. Edit the generated file under migrations/<timestamp>_add-foo-column.js
|
||
|
|
# - Put DDL in up() via pgm.sql(`ALTER TABLE ...`)
|
||
|
|
# - Write a real down() if rollback is safe; otherwise a throwing stub.
|
||
|
|
|
||
|
|
# 3. Apply locally against the dev DB (POSTGRES_URL from .env.local)
|
||
|
|
npm run migrate up
|
||
|
|
|
||
|
|
# 4. Update docs/SCHEMA_MAP.md to reflect the schema change.
|
||
|
|
|
||
|
|
# 5. Commit the migration file + docs together. CI runs lint + vitest
|
||
|
|
# only (D6); the migration itself isn't exercised in CI yet.
|
||
|
|
|
||
|
|
# 6. After merge + deploy, an operator runs the migration against prod:
|
||
|
|
# POSTGRES_URL=<prod> npm run migrate up
|
||
|
|
# (Or wait for the next setup-db invocation; it chains migrate up.)
|
||
|
|
```
|
||
|
|
|
||
|
|
### Onboarding a new env
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# 1. Clone + install
|
||
|
|
git clone <repo>
|
||
|
|
cd tcg-vault
|
||
|
|
npm install
|
||
|
|
|
||
|
|
# 2. Seed .env.local (POSTGRES_URL, JWT_SECRET, ADMIN_INITIAL_PASSWORD).
|
||
|
|
|
||
|
|
# 3. Run setup-db — chains migrate up, then seeds the admin user.
|
||
|
|
npm run setup-db
|
||
|
|
```
|
||
|
|
|
||
|
|
### Re-running setup against an existing env
|
||
|
|
|
||
|
|
Idempotent on both halves:
|
||
|
|
- The initial backfill migration uses `CREATE TABLE IF NOT EXISTS`,
|
||
|
|
so it's a no-op on tables that already exist.
|
||
|
|
- The admin-row INSERT uses `ON CONFLICT (email) DO NOTHING`.
|
||
|
|
|
||
|
|
The operator caveat from `drop-public-setup` Brief 1 still applies:
|
||
|
|
re-running setup-db does NOT rotate an existing admin row's password.
|
||
|
|
See README § "First-time admin setup" → operator-rotation note.
|
||
|
|
|
||
|
|
### Recovering from a failed migration
|
||
|
|
|
||
|
|
If `npm run migrate up` fails partway through (unlikely for the initial
|
||
|
|
backfill since each CREATE is independent; possible for future
|
||
|
|
migrations with multi-statement up()s), the operator's options:
|
||
|
|
|
||
|
|
1. **Fix forward**: edit the failing migration file, re-run
|
||
|
|
`npm run migrate up`. node-pg-migrate's default
|
||
|
|
`--single-transaction true` flag means a failure rolls back the
|
||
|
|
transaction, so the DB is left in the pre-migration state and the
|
||
|
|
`pgmigrations` row is NOT recorded. Re-running picks up cleanly.
|
||
|
|
|
||
|
|
2. **Skip a broken migration** (last resort): `npm run migrate up -- --fake`
|
||
|
|
marks pending migrations as applied without running them. Use this
|
||
|
|
ONLY if the schema state is already correct outside the tool's view
|
||
|
|
(e.g., the migration was applied manually via psql). Document any
|
||
|
|
`--fake` use in the convoy/PR that caused it.
|
||
|
|
|
||
|
|
## Follow-ups
|
||
|
|
|
||
|
|
- **`wire-migrate-into-ci`** (priority: P2 CI infra). Add a CI job that
|
||
|
|
runs `npm run migrate up` against a test DB (either a dedicated
|
||
|
|
Neon branch + secret, or a Postgres service container). Catches
|
||
|
|
syntactically-invalid migrations + most logical errors at PR time.
|
||
|
|
Deferred per D6.
|
||
|
|
- **`reconcile-historical-add-scripts`** (priority: P1 quality —
|
||
|
|
needed for fresh-env onboarding). Fold the effects of the 27
|
||
|
|
historical `scripts/add-*.js` / `scripts/fix-*.js` / `scripts/seed-*.js`
|
||
|
|
jobs into the migration history so a brand-new Neon branch can be
|
||
|
|
onboarded by `npm install` → `npm run setup-db` alone (without
|
||
|
|
manually replaying the historical scripts). Multi-PR: one
|
||
|
|
migration per logical change, ideally generated by reading the
|
||
|
|
scripts' SQL and re-shaping into idempotent `pgm.sql(...)` blocks
|
||
|
|
(with `IF NOT EXISTS` / `IF EXISTS` guards so re-application is
|
||
|
|
safe).
|
||
|
|
- **`retire-graveyard-scripts-after-audit`** (priority: P3 polish,
|
||
|
|
blocked on `reconcile-historical-add-scripts`). Once the migration
|
||
|
|
history captures all historical effects, the legacy `scripts/add-*.js`
|
||
|
|
/ `scripts/fix-*.js` / `scripts/seed-*.js` files can be deleted (or
|
||
|
|
moved to `scripts/historical/`). They remain no-go-zones until that
|
||
|
|
cleanup convoy lands.
|
||
|
|
- **`audit-node-pg-migrate-transitive-deps`** (priority: P3 hygiene).
|
||
|
|
`npm audit` reports 11 vulnerabilities (6 moderate, 5 high) coming
|
||
|
|
from node-pg-migrate's glob + yargs transitive deps. All in dev-only
|
||
|
|
paths; not exercised by node-pg-migrate's call sites. Surface only
|
||
|
|
if a security audit specifically flags this surface, or if
|
||
|
|
node-pg-migrate ships a v9 that updates the transitive tree.
|
||
|
|
- **`add-migration-template`** (priority: P3 DX). Add a custom template
|
||
|
|
via `--template-file-name` so generated migrations include the
|
||
|
|
project's preferred docstring shape + a reminder about
|
||
|
|
`docs/SCHEMA_MAP.md` updates. Surface if migration authoring proves
|
||
|
|
inconsistent.
|
||
|
|
|
||
|
|
## As-shipped
|
||
|
|
|
||
|
|
Shipped 2026-05-26 on `convoy/migration-tool`.
|
||
|
|
|
||
|
|
### Decisions ratified at gate 1
|
||
|
|
|
||
|
|
All seven decisions landed verbatim from the convoy spec's
|
||
|
|
recommendations. No mid-execution surprises that would have routed
|
||
|
|
back through an architect bounce.
|
||
|
|
|
||
|
|
### As-shipped surface
|
||
|
|
|
||
|
|
- **`migrations/` directory created at repo root** (was: empty / nonexistent).
|
||
|
|
- **`migrations/1779853647564_initial-schema.js`** added. ~155 lines.
|
||
|
|
Seven `pgm.sql(\`CREATE TABLE IF NOT EXISTS ...\`)` blocks
|
||
|
|
reproducing `scripts/setup-neon-db.js`'s 7-table DDL verbatim.
|
||
|
|
Down-migration is a hard stub that throws with a long-form error
|
||
|
|
message naming the alternative (Neon branch + forward-apply).
|
||
|
|
Lint-clean (no new ignore patterns in `eslint.config.mjs`).
|
||
|
|
- **`scripts/setup-neon-db.js`** refactored. The seven `await sql\`CREATE TABLE IF NOT EXISTS\`` blocks are removed (DDL now lives in the migration). A `runMigrations()` helper is added that spawns `npm run migrate up` via `node:child_process.spawn({ stdio: 'inherit', shell: false })` and rejects with a wrapped error on non-zero exit. A `POSTGRES_URL` pre-flight check is added (was previously implicit). The success/error message copy is updated to mention the migration step explicitly so the operator's mental model matches the new pipeline. The admin-row INSERT (with `ON CONFLICT (email) DO NOTHING`) and the `ADMIN_INITIAL_PASSWORD` env-var check are preserved verbatim.
|
||
|
|
- **`package.json`**: `scripts.migrate` added (`node-pg-migrate --database-url-var POSTGRES_URL --envPath .env.local --migrations-dir migrations --verbose`). `devDependencies` adds `node-pg-migrate@^8.0.4` + `pg@^8.21.0`.
|
||
|
|
- **`README.md`** updated: § Installation step 4 documents the migration chain; new § "Schema changes (post-`migration-tool` convoy)" explains the create/edit/up/commit flow; § "First-time admin setup" mentions the migrate step.
|
||
|
|
- **`AGENTS.md`** updated: § 3 Conventions gains a "Schema changes" bullet pointing at the new flow + `.convoys/migration-tool.md`; § 4 Gotcha #6 flipped from open → RESOLVED with the as-shipped paragraph (tool / dir / tracking-table / idempotency notes).
|
||
|
|
- **`.cursor/rules/no-go-zones.mdc`** updated: "Schema changes" rule rewritten to describe the `node-pg-migrate` flow; legacy `scripts/migrations/YYYY-MM-DD-<slug>.js` convention documented as "preserved for the lone existing pre-tool migration; not used for new work".
|
||
|
|
- **`.cursor/rules/db-and-schema.mdc`** updated: § "Schema source of truth" rewritten to point at `migrations/` and the `npm run migrate create ...` workflow.
|
||
|
|
- **`docs/SCHEMA_MAP.md`** updated: preamble re-scopes the file (tool-managed schema lives in `migrations/`; this file remains the curated reference for the full prod shape including historical add-*.js columns); § Regeneration updated.
|
||
|
|
|
||
|
|
### Verification (all gates green pre-PR)
|
||
|
|
|
||
|
|
- `npm run lint` → exit 1 with **128 problems** (baseline preserved, zero regression). The new migration file is lint-clean; no new ignore patterns added.
|
||
|
|
- `npm run test:run` → **21/21 pass** in ~1.3s (4 test files, no rate-limit / migration touchpoint).
|
||
|
|
- `node --check migrations/1779853647564_initial-schema.js` → exit 0.
|
||
|
|
- `node --check scripts/setup-neon-db.js` → exit 0.
|
||
|
|
- Module-load + `down()` throw verification:
|
||
|
|
`node -e "import('./migrations/1779853647564_initial-schema.js').then(m => m.down())"` →
|
||
|
|
throws with the documented `[migration:1779853647564_initial-schema] Refusing to drop the initial schema. ...` message.
|
||
|
|
- `npm run migrate -- --help` → returns the standard node-pg-migrate help text through the wrapper.
|
||
|
|
|
||
|
|
### Live verification status
|
||
|
|
|
||
|
|
**Deferred** per the convoy spec. The parent did not have a throwaway
|
||
|
|
Neon branch available. The operator's optional post-merge verification
|
||
|
|
sequence is documented in § Operator runbook → "Re-running setup
|
||
|
|
against an existing env".
|
||
|
|
|
||
|
|
### Operator action required going forward
|
||
|
|
|
||
|
|
**None for the convoy itself.** The migration is idempotent against
|
||
|
|
the existing prod schema (CREATE TABLE IF NOT EXISTS no-ops on
|
||
|
|
existing tables; the `pgmigrations` row is the only DB-side change).
|
||
|
|
No new env vars; no new secrets. The existing `POSTGRES_URL` +
|
||
|
|
`ADMIN_INITIAL_PASSWORD` contract is preserved.
|
||
|
|
|
||
|
|
**Optional but recommended post-merge:**
|
||
|
|
|
||
|
|
1. The next deploy that runs `setup-neon-db.js` will silently apply
|
||
|
|
the backfill migration (recording it in `pgmigrations`). No
|
||
|
|
operator action; this is just-in-time chained.
|
||
|
|
|
||
|
|
2. To pre-apply the migration without re-running the seed step:
|
||
|
|
`POSTGRES_URL=<...> npm run migrate up`.
|
||
|
|
|
||
|
|
### Cross-validation finding
|
||
|
|
|
||
|
|
Same shape as the `add-rate-limiting` cross-validation finding: the
|
||
|
|
existing `Playwright smoke` 3/3 spec defends the post-migration-tool
|
||
|
|
deployment without anyone writing a dedicated test. Smoke calls
|
||
|
|
`/api/health` once per run; `/api/health` reads from the `users` table
|
||
|
|
(via `lib/database.js` health-check query — verify with the smoke run
|
||
|
|
on the PR) which is created by both the initial backfill AND the
|
||
|
|
pre-existing `scripts/setup-neon-db.js` DDL, so smoke's signal is
|
||
|
|
invariant to which mechanism populated the schema. Sixth consecutive
|
||
|
|
convoy where the same 3-test smoke spec defends the auth surface
|
||
|
|
through a sweeping change (PR #15 Layout default-user → PR #19
|
||
|
|
CORS-tighten → PR #20 rate-limiting → PR #21 pick-a-name → PR #25
|
||
|
|
reset-db-fix → this convoy).
|
||
|
|
|
||
|
|
### What did NOT change
|
||
|
|
|
||
|
|
- `lib/database.js`, `lib/permission-middleware.js`, `lib/auth-secret.js`, `lib/rate-limit.js` — no auth or runtime surface touched.
|
||
|
|
- `pages/api/**/*.js` — no API handlers touched.
|
||
|
|
- `test/**` — no test surface touched (Decision-6-equivalent: per-route handler tests for the migration runner would be valuable but are out of scope; the migration's correctness is verified via the operator runbook's manual sequence).
|
||
|
|
- `scripts/migrations/2026-05-24-rename-admin-email.js` — preserved verbatim; not migrated into the new `migrations/` directory because it's already-applied history (the no-go-zones rule covers it).
|
||
|
|
- `.github/workflows/**` — no new CI job (Decision 6 defers `wire-migrate-into-ci` to a follow-up convoy).
|
||
|
|
- `package-lock.json` semantics — only adds the new transitive deps; no version bumps to existing deps.
|
||
|
|
- The legacy 27 `scripts/add-*.js` / `scripts/fix-*.js` / `scripts/seed-*.js` files — all preserved as append-only history per the no-go-zones rule.
|
||
|
|
|
||
|
|
### Subagent / multitask footnote
|
||
|
|
|
||
|
|
This convoy ran with the parent owning architecture + implementation
|
||
|
|
end-to-end (no architect or implementer subagent dispatch). The
|
||
|
|
convoy spec pre-ratified each Decision's recommended path, and the
|
||
|
|
implementation surface was a small set of well-bounded file edits
|
||
|
|
following the spec's "The change (implementation shape)" checklist
|
||
|
|
verbatim. A mid-execution worktree migration was required (the parent
|
||
|
|
opened the convoy from the main worktree, which was incidentally
|
||
|
|
checked out on a sibling branch `convoy/purge-weak-creds-from-helpers`
|
||
|
|
due to a parallel agent's activity; the parent stashed its work,
|
||
|
|
created `tcg-vault-worktrees/migration-tool` as a fresh worktree on
|
||
|
|
the canonical `convoy/migration-tool` branch, re-applied the work
|
||
|
|
there via a tracked-changes patch + an untracked-files tar, and
|
||
|
|
continued cleanly). No file content was lost in that migration; the
|
||
|
|
git history is single-branch from the worktree's perspective.
|