deckhearth/docs/MIGRATION_VERIFICATION_RUNBOOK.md

Migration verification runbook

How to confirm that a fresh Neon branch's migrated schema matches a long-lived environment (prod, preview, or similar). The migration history under migrations/ is authoritative; this runbook is the manual check until the queued wire-migrate-into-ci convoy automates it on every PR.

When to run

• After any reconcile-adjacent convoy lands on main.
• Before releasing schema changes to prod when migration correctness is uncertain.
• When suspecting drift between an env's actual schema and the migration history (rare).

Prerequisites

• A Neon account with permission to create and delete branches (or any other way to spin up a fresh Postgres DB on the same major version as prod).
• POSTGRES_URL for the branch under test — typically stored in a local .env.local.branch or passed inline (never commit branch URLs).
• ADMIN_INITIAL_PASSWORD set (required by npm run setup-db; see AGENTS.md § 5).
• pg_dump installed locally (PostgreSQL 16+), or available via a Docker container with network access to Neon.

Procedure

1. Create a throwaway Neon branch

neon branches create --name migrate-verify-YYYYMMDD --parent main

Or create an equivalent branch via the Neon console. Copy the branch connection string into POSTGRES_URL.

2. Onboard the fresh branch via setup-db

POSTGRES_URL=<branch-url> \
ADMIN_INITIAL_PASSWORD=$(openssl rand -base64 24) \
  npm run setup-db

setup-db runs npm run migrate up (every file in migrations/ in timestamp order) and seeds the admin user.

3. Snapshot the fresh branch schema

POSTGRES_URL=<branch-url> pg_dump --schema-only --no-owner --no-acl \
  --schema=public > fresh-schema.sql

4. Snapshot prod schema (read-only)

POSTGRES_URL=<prod-url> pg_dump --schema-only --no-owner --no-acl \
  --schema=public > prod-schema.sql

Use a read-only role or connection if your operator policy requires it. Do not run DDL against prod during verification.

5. Diff

diff -u prod-schema.sql fresh-schema.sql

• Empty output — structural parity confirmed for tables, columns, constraints, and indexes in public.
• Non-empty output — investigate before releasing; see below.

6. Delete the throwaway branch

neon branches delete migrate-verify-YYYYMMDD

Known-safe diffs

These differences from pg_dump are not real schema drift:

• Schema owner / comment lines that differ between Neon projects.
• Extension version pins (CREATE EXTENSION version strings).
• pgmigrations row content — the fresh env accumulates rows as migrations apply; prod may have the same rows with different apply timestamps.
• Constraint or index name differences when prod objects were created by historical scripts/add-* jobs with autogenerated names and fresh envs use migration-defined names (material shape must still match).
• Column order within a table (historical ALTER order vs migration order).

When drift is detected

1. Halt — do not release until the delta is understood.
2. Identify which env is canonical. Usually prod. If prod is missing a column that exists in the migration history, run POSTGRES_URL=<prod-url> npm run migrate up to catch up.
3. If prod has something not in the migration history, file a new convoy or brief describing the delta and ship a reconciliation migration following the pattern in .convoys/reconcile-historical-add-scripts/.
4. Never edit existing migrations — pgmigrations pins applied files. Always add a new dated migration to correct schema.

Supplementary spot-checks

For high-risk tables, compare column / constraint / index counts between envs:

SELECT table_name, column_name, data_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_schema = 'public'
ORDER BY table_name, ordinal_position;

SELECT table_name, constraint_name, constraint_type
FROM information_schema.table_constraints
WHERE table_schema = 'public'
ORDER BY table_name, constraint_name;

SELECT tablename, indexname, indexdef
FROM pg_indexes
WHERE schemaname = 'public'
ORDER BY tablename, indexname;

A mismatch in column count, constraint count, or index count is material drift.

Automation path

The queued wire-migrate-into-ci convoy (see .convoys/ship-readiness.md § Queued convoys) will run npm run migrate up against a test DB on every PR. Until that ships, this operator runbook is the only automated-parity signal.

Cross-references

• AGENTS.md § 4 Gotcha #6 — migration tool adoption history.
• .convoys/migration-tool.md — node-pg-migrate adoption convoy.
• .convoys/reconcile-historical-add-scripts.md § As-shipped — the convoy that folded historical add-script DDL into migrations/.
• .cursor/rules/db-and-schema.mdc — schema-change conventions.