deckhearth/docs/MIGRATION_VERIFICATION_RUNBOOK.md

# Migration verification runbook

How to confirm that a fresh Neon branch's migrated schema matches a
long-lived environment (prod, preview, or similar). The migration history
under `migrations/` is authoritative; this runbook is the manual check until
the queued `wire-migrate-into-ci` convoy automates it on every PR.

## When to run

- After any reconcile-adjacent convoy lands on `main`.
- Before releasing schema changes to prod when migration correctness is
  uncertain.
- When suspecting drift between an env's actual schema and the migration
  history (rare).

## Prerequisites

- A Neon account with permission to create and delete branches (or any
  other way to spin up a fresh Postgres DB on the same major version as
  prod).
- `POSTGRES_URL` for the branch under test — typically stored in a local
  `.env.local.branch` or passed inline (never commit branch URLs).
- `ADMIN_INITIAL_PASSWORD` set (required by `npm run setup-db`; see
  `AGENTS.md` § 5).
- `pg_dump` installed locally (PostgreSQL 16+), or available via a Docker
  container with network access to Neon.

## Procedure

### 1. Create a throwaway Neon branch

```bash
neon branches create --name migrate-verify-YYYYMMDD --parent main
```

Or create an equivalent branch via the Neon console. Copy the branch
connection string into `POSTGRES_URL`.

### 2. Onboard the fresh branch via setup-db

```bash
POSTGRES_URL=<branch-url> \
ADMIN_INITIAL_PASSWORD=$(openssl rand -base64 24) \
  npm run setup-db
```

`setup-db` runs `npm run migrate up` (every file in `migrations/` in
timestamp order) and seeds the admin user.

### 3. Snapshot the fresh branch schema

```bash
POSTGRES_URL=<branch-url> pg_dump --schema-only --no-owner --no-acl \
  --schema=public > fresh-schema.sql
```

### 4. Snapshot prod schema (read-only)

```bash
POSTGRES_URL=<prod-url> pg_dump --schema-only --no-owner --no-acl \
  --schema=public > prod-schema.sql
```

Use a read-only role or connection if your operator policy requires it.
Do not run DDL against prod during verification.

### 5. Diff

```bash
diff -u prod-schema.sql fresh-schema.sql
```

- **Empty output** — structural parity confirmed for tables, columns,
  constraints, and indexes in `public`.
- **Non-empty output** — investigate before releasing; see below.

### 6. Delete the throwaway branch

```bash
neon branches delete migrate-verify-YYYYMMDD
```

## Known-safe diffs

These differences from `pg_dump` are **not** real schema drift:

- Schema owner / comment lines that differ between Neon projects.
- Extension version pins (`CREATE EXTENSION` version strings).
- `pgmigrations` row content — the fresh env accumulates rows as
  migrations apply; prod may have the same rows with different apply
  timestamps.
- Constraint or index **name** differences when prod objects were created
  by historical `scripts/add-*` jobs with autogenerated names and fresh
  envs use migration-defined names (material shape must still match).
- Column **order** within a table (historical ALTER order vs migration
  order).

## When drift is detected

1. **Halt** — do not release until the delta is understood.
2. **Identify which env is canonical.** Usually prod. If prod is missing
   a column that exists in the migration history, run
   `POSTGRES_URL=<prod-url> npm run migrate up` to catch up.
3. **If prod has something not in the migration history**, file a new
   convoy or brief describing the delta and ship a reconciliation
   migration following the pattern in
   `.convoys/reconcile-historical-add-scripts/`.
4. **Never edit existing migrations** — `pgmigrations` pins applied
   files. Always add a new dated migration to correct schema.

## Supplementary spot-checks

For high-risk tables, compare column / constraint / index counts between
envs:

```sql
SELECT table_name, column_name, data_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_schema = 'public'
ORDER BY table_name, ordinal_position;

SELECT table_name, constraint_name, constraint_type
FROM information_schema.table_constraints
WHERE table_schema = 'public'
ORDER BY table_name, constraint_name;

SELECT tablename, indexname, indexdef
FROM pg_indexes
WHERE schemaname = 'public'
ORDER BY tablename, indexname;
```

A mismatch in column count, constraint count, or index count is material
drift.

## Automation path

The queued `wire-migrate-into-ci` convoy (see `.convoys/ship-readiness.md`
§ Queued convoys) will run `npm run migrate up` against a test DB on every
PR. Until that ships, this operator runbook is the only automated-parity
signal.

## Cross-references

- `AGENTS.md` § 4 Gotcha #6 — migration tool adoption history.
- `.convoys/migration-tool.md` — `node-pg-migrate` adoption convoy.
- `.convoys/reconcile-historical-add-scripts.md` § As-shipped — the convoy
  that folded historical add-script DDL into `migrations/`.
- `.cursor/rules/db-and-schema.mdc` — schema-change conventions.