deckhearth/.convoys/catalog-sync-vercel-cron.md

---
name: catalog-sync-vercel-cron
classification: infra-only
success_metric: |
  New MTG and Pokémon sets appear in the cards catalog within one week of
  upstream API availability without manual admin import; each cron run logs
  imported/skipped set counts and fails loud on errors.
skip:
  - ia
  - ux
  - visual
  - a11y
  - design
status: in-progress
created: 2026-05-27
depends_on:
  - redesign-scanner-flow
  - scanner-correctness-polish
  - add-real-ocr-layer
blocked_by_policy: |
  Unblocked 2026-05-27 after scanner pipeline + audit follow-ups merged.
---

# Convoy: catalog-sync-vercel-cron

Scheduled catalog freshness via **Vercel Cron** (not GitHub Actions — operator
preference: already on Vercel paid plan; avoids GitHub Actions minute limits).

## Why

The Perfect Order Seel scan failure (2026-05-27) exposed a catalog gap: Layer 1
matched the name "Seel" against nine *old* printings because **Perfect Order is
not in the database**. `card_submissions` now bridges unknown cards for admin
review, but scanning still degrades until new sets are imported.

Today catalog updates are **fully manual**:

- Admin UI at `/admin/card-import` (MTG + Pokémon only; set code typed by hand)
- One-off scripts (`import-popular-sets.js`, `bulk-import-all.js`) with **static**
  set lists
- Lorcana import uses a **hardcoded** `setCodeMap` in `import-lorcana.js`

There is no scheduled job. `scripts/README.md` documents "import new sets as they
release" as the ongoing process — easy to forget.

## Operator decision (2026-05-27)

- **Scheduler:** Vercel Cron hitting a protected API route on the production
  deployment (or a dedicated Preview with prod DB — architect decides at gate-1).
- **Not GitHub Actions cron** — operator prefers Vercel to stay within GitHub
  Actions free-tier limits.
- **Timing:** Build **after** scanner pipeline convoys land (see `blocked_by_policy`
  above). Scanner correctness + UX (#4–#6 in the scanner audit portfolio) take
  priority.

## Scope

### In scope

1. **Extract shared import logic** from `pages/api/cards/import-mtg.js` and
   `pages/api/cards/import-pokemon.js` into `lib/card-import/` (or similar) so
   cron, admin UI, and scripts call one code path. Idempotent skips preserved
   (MTG: `scryfall_id`; Pokémon: existing duplicate checks).

2. **Set discovery (delta sync)**
   - **MTG:** Scryfall `GET /sets` — compare `code` + release date against
     `SELECT DISTINCT set_code FROM cards WHERE game = 'MTG'`.
   - **Pokémon:** Pokémon TCG API `GET /v2/sets` — compare `id` against catalog;
     filter to sets released in the last N days or not yet present in DB.
   - **Lorcana:** Out of scope for v1 automation unless Lorcast set list is
     fetched dynamically; v1 may log "manual Lorcana map update required" and skip.

3. **Protected cron endpoint** — e.g. `GET /api/cron/sync-catalog` or
   `POST /api/admin/sync-catalog`:
   - Authenticate via `CRON_SECRET` header (Vercel Cron
     [securing cron jobs](https://vercel.com/docs/cron-jobs/manage-cron-jobs#securing-cron-jobs)
     pattern) — **not** JWT admin session.
   - Reuse or bypass `checkImportRateLimit` thoughtfully: cron is a single
     system actor; may need a dedicated limiter class or internal-only bypass with
     hard cap on sets per run (e.g. max 3 sets/run, 1s delay between sets).
   - Never expose unauthenticated bulk INSERT into `cards`.

4. **`vercel.json` cron schedule** — weekly default (e.g. `0 6 * * 1` UTC);
   `workflow_dispatch`-equivalent: manual hit with `CRON_SECRET` for on-demand runs.

5. **Observability**
   - Minimum: structured console log + HTTP 200 body with `{ imported, skipped,
     errors, setsProcessed }`.
   - Nice-to-have (v1.1): `catalog_sync_runs` migration (`started_at`, `finished_at`,
     `sets_imported`, `error_json`).

6. **Docs** — update `scripts/README.md` § "For Ongoing Management" to point at
   cron + manual override via admin UI.

### Out of scope (v1)

- Auto-promoting `card_submissions` when a matching set import completes (follow-up
  convoy `reconcile-submissions-after-catalog-sync`).
- Hourly sync (weekly is sufficient for TCG release cadence).
- Full `bulk-import-all.js` replacement or re-import of historical sets.
- GitHub Actions scheduled workflow (explicitly rejected by operator).
- Running import jobs ad-hoc against prod without pacing (AGENTS.md no-go: rate limits).

## Proposed architecture

```
Vercel Cron (weekly)
  → GET /api/cron/sync-catalog  (+ Authorization: Bearer $CRON_SECRET)
    → discoverNewSets('mtg' | 'pokemon')
    → for each missing set (max N per run):
        → importSetFromScryfall(code) / importSetFromPokemonTcg(id)
        → delay 1–3s (respect upstream + existing import rate limits)
    → return summary JSON
```

**Env vars (new):**

| Var | Purpose |
| --- | --- |
| `CRON_SECRET` | Vercel Cron auth header; rotate via Vercel dashboard |
| `POKEMON_TCG_API_KEY` | If not already set — Pokémon API key for set discovery |

**Existing vars reused:** `POSTGRES_URL`, Scryfall needs no key.

## Roles invoked

1. `role-architect` — gate-1: cron auth shape, rate-limit policy, Lorcana v1 stance,
   sets-per-run cap.
2. `role-implementer` — brief 1 (lib extract + cron route + vercel.json); brief 2
   (discovery + docs) if split.
3. `role-reviewer` — post-PR.

## Todos

- [x] Extract `lib/card-import/mtg.js` + `lib/card-import/pokemon.js`
- [x] Implement set discovery + delta diff
- [x] Add `/api/cron/sync-catalog` + `vercel.json` cron entry
- [x] Document operator setup (`CRON_SECRET`, manual trigger, monitoring) — scripts/README.md
- [ ] Architect: ratify cron auth, import rate-limit bypass/cap, schedule cadence (defaults shipped)
- [ ] Smoke: one dry-run against staging Neon branch (operator)

## Operator action required (at ship time)

1. Set `CRON_SECRET` in Vercel project env (generate: `openssl rand -base64 32`).
2. Confirm Pokémon TCG API key is present if set discovery uses authenticated endpoints.
3. After first cron run, spot-check Vercel function logs + `cards` row count for a
   known recent set.
4. Optional: alert on cron failure (Vercel log drain / email) — not required for v1.

## Relationship to scanner work

| Scanner deliverable | How catalog sync helps |
| --- | --- |
| `card_submissions` queue (shipped) | Safety net when sync hasn't run yet |
| Disambiguation + "not listed" (in progress) | UX when catalog is stale |
| **This convoy** | Reduces stale-catalog frequency at the source |

Queue **after** `redesign-scanner-flow`, `scanner-correctness-polish`, and
`rename-collections-vocabulary` unless operator reprioritizes.

## Follow-up convoys (not v1)

- **`reconcile-submissions-after-catalog-sync`** — when a set import lands, auto-match
  pending `card_submissions` with matching `ocr_payload` set/name/number.
- **`lorcana-dynamic-set-discovery`** — replace hardcoded `setCodeMap` in
  `import-lorcana.js`.
- **`catalog-sync-runs-table`** — migration for audit trail if console logs prove
  insufficient.

## Test plan

- Unit: set-diff logic (mock DB rows vs mock API set list).
- Integration (staging): cron endpoint with `CRON_SECRET` imports one known small set;
  second run skips all (idempotent).
- Manual: verify admin `/admin/card-import` still works after lib extraction.