deckhearth/.convoys/catalog-sync-vercel-cron.md

name	classification	success_metric	skip	status	created	depends_on	blocked_by_policy
catalog-sync-vercel-cron	infra-only	New MTG and Pokémon sets appear in the cards catalog within one week of upstream API availability without manual admin import; each cron run logs imported/skipped set counts and fails loud on errors.	ia ux visual a11y design	open	2026-05-27	redesign-scanner-flow scanner-correctness-polish add-real-ocr-layer	Operator requested finishing the scanner pipeline and other in-flight convoys before starting this work. Do not pick up until those are merged or explicitly reprioritized.

Convoy: catalog-sync-vercel-cron

Scheduled catalog freshness via Vercel Cron (not GitHub Actions — operator preference: already on Vercel paid plan; avoids GitHub Actions minute limits).

Why

The Perfect Order Seel scan failure (2026-05-27) exposed a catalog gap: Layer 1 matched the name "Seel" against nine old printings because Perfect Order is not in the database. card_submissions now bridges unknown cards for admin review, but scanning still degrades until new sets are imported.

Today catalog updates are fully manual:

• Admin UI at /admin/card-import (MTG + Pokémon only; set code typed by hand)
• One-off scripts (import-popular-sets.js, bulk-import-all.js) with static set lists
• Lorcana import uses a hardcoded setCodeMap in import-lorcana.js

There is no scheduled job. scripts/README.md documents "import new sets as they release" as the ongoing process — easy to forget.

Operator decision (2026-05-27)

• Scheduler: Vercel Cron hitting a protected API route on the production deployment (or a dedicated Preview with prod DB — architect decides at gate-1).
• Not GitHub Actions cron — operator prefers Vercel to stay within GitHub Actions free-tier limits.
• Timing: Build after scanner pipeline convoys land (see blocked_by_policy above). Scanner correctness + UX (#4–#6 in the scanner audit portfolio) take priority.

Scope

In scope

1. Extract shared import logic from pages/api/cards/import-mtg.js and pages/api/cards/import-pokemon.js into lib/card-import/ (or similar) so cron, admin UI, and scripts call one code path. Idempotent skips preserved (MTG: scryfall_id; Pokémon: existing duplicate checks).

2. Set discovery (delta sync)

   • MTG: Scryfall GET /sets — compare code + release date against SELECT DISTINCT set_code FROM cards WHERE game = 'MTG'.
   • Pokémon: Pokémon TCG API GET /v2/sets — compare id against catalog; filter to sets released in the last N days or not yet present in DB.
   • Lorcana: Out of scope for v1 automation unless Lorcast set list is fetched dynamically; v1 may log "manual Lorcana map update required" and skip.

3. Protected cron endpoint — e.g. GET /api/cron/sync-catalog or POST /api/admin/sync-catalog:

   • Authenticate via CRON_SECRET header (Vercel Cron securing cron jobs pattern) — not JWT admin session.
   • Reuse or bypass checkImportRateLimit thoughtfully: cron is a single system actor; may need a dedicated limiter class or internal-only bypass with hard cap on sets per run (e.g. max 3 sets/run, 1s delay between sets).
   • Never expose unauthenticated bulk INSERT into cards.

4. vercel.json cron schedule — weekly default (e.g. 0 6 * * 1 UTC); workflow_dispatch-equivalent: manual hit with CRON_SECRET for on-demand runs.

5. Observability

   • Minimum: structured console log + HTTP 200 body with { imported, skipped, errors, setsProcessed }.
   • Nice-to-have (v1.1): catalog_sync_runs migration (started_at, finished_at, sets_imported, error_json).

6. Docs — update scripts/README.md § "For Ongoing Management" to point at cron + manual override via admin UI.

Out of scope (v1)

• Auto-promoting card_submissions when a matching set import completes (follow-up convoy reconcile-submissions-after-catalog-sync).
• Hourly sync (weekly is sufficient for TCG release cadence).
• Full bulk-import-all.js replacement or re-import of historical sets.
• GitHub Actions scheduled workflow (explicitly rejected by operator).
• Running import jobs ad-hoc against prod without pacing (AGENTS.md no-go: rate limits).

Proposed architecture

Vercel Cron (weekly)
  → GET /api/cron/sync-catalog  (+ Authorization: Bearer $CRON_SECRET)
    → discoverNewSets('mtg' | 'pokemon')
    → for each missing set (max N per run):
        → importSetFromScryfall(code) / importSetFromPokemonTcg(id)
        → delay 1–3s (respect upstream + existing import rate limits)
    → return summary JSON

Env vars (new):

Var	Purpose
CRON_SECRET	Vercel Cron auth header; rotate via Vercel dashboard
POKEMON_TCG_API_KEY	If not already set — Pokémon API key for set discovery

Existing vars reused: POSTGRES_URL, Scryfall needs no key.

Roles invoked

1. role-architect — gate-1: cron auth shape, rate-limit policy, Lorcana v1 stance, sets-per-run cap.
2. role-implementer — brief 1 (lib extract + cron route + vercel.json); brief 2 (discovery + docs) if split.
3. role-reviewer — post-PR.

Todos

• Architect: ratify cron auth, import rate-limit bypass/cap, schedule cadence
• Extract lib/card-import/mtg.js + lib/card-import/pokemon.js
• Implement set discovery + delta diff
• Add /api/cron/sync-catalog + vercel.json cron entry
• Document operator setup (CRON_SECRET, manual trigger, monitoring)
• Smoke: one dry-run against staging Neon branch

Operator action required (at ship time)

1. Set CRON_SECRET in Vercel project env (generate: openssl rand -base64 32).
2. Confirm Pokémon TCG API key is present if set discovery uses authenticated endpoints.
3. After first cron run, spot-check Vercel function logs + cards row count for a known recent set.
4. Optional: alert on cron failure (Vercel log drain / email) — not required for v1.

Relationship to scanner work

Scanner deliverable	How catalog sync helps
card_submissions queue (shipped)	Safety net when sync hasn't run yet
Disambiguation + "not listed" (in progress)	UX when catalog is stale
This convoy	Reduces stale-catalog frequency at the source

Queue after redesign-scanner-flow, scanner-correctness-polish, and rename-collections-vocabulary unless operator reprioritizes.

Follow-up convoys (not v1)

• reconcile-submissions-after-catalog-sync — when a set import lands, auto-match pending card_submissions with matching ocr_payload set/name/number.
• lorcana-dynamic-set-discovery — replace hardcoded setCodeMap in import-lorcana.js.
• catalog-sync-runs-table — migration for audit trail if console logs prove insufficient.

Test plan

• Unit: set-diff logic (mock DB rows vs mock API set list).
• Integration (staging): cron endpoint with CRON_SECRET imports one known small set; second run skips all (idempotent).
• Manual: verify admin /admin/card-import still works after lib extraction.