diff --git a/.convoys/catalog-sync-vercel-cron.md b/.convoys/catalog-sync-vercel-cron.md new file mode 100644 index 0000000..3d005b9 --- /dev/null +++ b/.convoys/catalog-sync-vercel-cron.md @@ -0,0 +1,177 @@ +--- +name: catalog-sync-vercel-cron +classification: infra-only +success_metric: | + New MTG and Pokémon sets appear in the cards catalog within one week of + upstream API availability without manual admin import; each cron run logs + imported/skipped set counts and fails loud on errors. +skip: + - ia + - ux + - visual + - a11y + - design +status: open +created: 2026-05-27 +depends_on: + - redesign-scanner-flow + - scanner-correctness-polish + - add-real-ocr-layer +blocked_by_policy: | + Operator requested finishing the scanner pipeline and other in-flight convoys + before starting this work. Do not pick up until those are merged or explicitly + reprioritized. +--- + +# Convoy: catalog-sync-vercel-cron + +Scheduled catalog freshness via **Vercel Cron** (not GitHub Actions — operator +preference: already on Vercel paid plan; avoids GitHub Actions minute limits). + +## Why + +The Perfect Order Seel scan failure (2026-05-27) exposed a catalog gap: Layer 1 +matched the name "Seel" against nine *old* printings because **Perfect Order is +not in the database**. `card_submissions` now bridges unknown cards for admin +review, but scanning still degrades until new sets are imported. + +Today catalog updates are **fully manual**: + +- Admin UI at `/admin/card-import` (MTG + Pokémon only; set code typed by hand) +- One-off scripts (`import-popular-sets.js`, `bulk-import-all.js`) with **static** + set lists +- Lorcana import uses a **hardcoded** `setCodeMap` in `import-lorcana.js` + +There is no scheduled job. `scripts/README.md` documents "import new sets as they +release" as the ongoing process — easy to forget. + +## Operator decision (2026-05-27) + +- **Scheduler:** Vercel Cron hitting a protected API route on the production + deployment (or a dedicated Preview with prod DB — architect decides at gate-1). +- **Not GitHub Actions cron** — operator prefers Vercel to stay within GitHub + Actions free-tier limits. +- **Timing:** Build **after** scanner pipeline convoys land (see `blocked_by_policy` + above). Scanner correctness + UX (#4–#6 in the scanner audit portfolio) take + priority. + +## Scope + +### In scope + +1. **Extract shared import logic** from `pages/api/cards/import-mtg.js` and + `pages/api/cards/import-pokemon.js` into `lib/card-import/` (or similar) so + cron, admin UI, and scripts call one code path. Idempotent skips preserved + (MTG: `scryfall_id`; Pokémon: existing duplicate checks). + +2. **Set discovery (delta sync)** + - **MTG:** Scryfall `GET /sets` — compare `code` + release date against + `SELECT DISTINCT set_code FROM cards WHERE game = 'MTG'`. + - **Pokémon:** Pokémon TCG API `GET /v2/sets` — compare `id` against catalog; + filter to sets released in the last N days or not yet present in DB. + - **Lorcana:** Out of scope for v1 automation unless Lorcast set list is + fetched dynamically; v1 may log "manual Lorcana map update required" and skip. + +3. **Protected cron endpoint** — e.g. `GET /api/cron/sync-catalog` or + `POST /api/admin/sync-catalog`: + - Authenticate via `CRON_SECRET` header (Vercel Cron + [securing cron jobs](https://vercel.com/docs/cron-jobs/manage-cron-jobs#securing-cron-jobs) + pattern) — **not** JWT admin session. + - Reuse or bypass `checkImportRateLimit` thoughtfully: cron is a single + system actor; may need a dedicated limiter class or internal-only bypass with + hard cap on sets per run (e.g. max 3 sets/run, 1s delay between sets). + - Never expose unauthenticated bulk INSERT into `cards`. + +4. **`vercel.json` cron schedule** — weekly default (e.g. `0 6 * * 1` UTC); + `workflow_dispatch`-equivalent: manual hit with `CRON_SECRET` for on-demand runs. + +5. **Observability** + - Minimum: structured console log + HTTP 200 body with `{ imported, skipped, + errors, setsProcessed }`. + - Nice-to-have (v1.1): `catalog_sync_runs` migration (`started_at`, `finished_at`, + `sets_imported`, `error_json`). + +6. **Docs** — update `scripts/README.md` § "For Ongoing Management" to point at + cron + manual override via admin UI. + +### Out of scope (v1) + +- Auto-promoting `card_submissions` when a matching set import completes (follow-up + convoy `reconcile-submissions-after-catalog-sync`). +- Hourly sync (weekly is sufficient for TCG release cadence). +- Full `bulk-import-all.js` replacement or re-import of historical sets. +- GitHub Actions scheduled workflow (explicitly rejected by operator). +- Running import jobs ad-hoc against prod without pacing (AGENTS.md no-go: rate limits). + +## Proposed architecture + +``` +Vercel Cron (weekly) + → GET /api/cron/sync-catalog (+ Authorization: Bearer $CRON_SECRET) + → discoverNewSets('mtg' | 'pokemon') + → for each missing set (max N per run): + → importSetFromScryfall(code) / importSetFromPokemonTcg(id) + → delay 1–3s (respect upstream + existing import rate limits) + → return summary JSON +``` + +**Env vars (new):** + +| Var | Purpose | +| --- | --- | +| `CRON_SECRET` | Vercel Cron auth header; rotate via Vercel dashboard | +| `POKEMON_TCG_API_KEY` | If not already set — Pokémon API key for set discovery | + +**Existing vars reused:** `POSTGRES_URL`, Scryfall needs no key. + +## Roles invoked + +1. `role-architect` — gate-1: cron auth shape, rate-limit policy, Lorcana v1 stance, + sets-per-run cap. +2. `role-implementer` — brief 1 (lib extract + cron route + vercel.json); brief 2 + (discovery + docs) if split. +3. `role-reviewer` — post-PR. + +## Todos + +- [ ] Architect: ratify cron auth, import rate-limit bypass/cap, schedule cadence +- [ ] Extract `lib/card-import/mtg.js` + `lib/card-import/pokemon.js` +- [ ] Implement set discovery + delta diff +- [ ] Add `/api/cron/sync-catalog` + `vercel.json` cron entry +- [ ] Document operator setup (`CRON_SECRET`, manual trigger, monitoring) +- [ ] Smoke: one dry-run against staging Neon branch + +## Operator action required (at ship time) + +1. Set `CRON_SECRET` in Vercel project env (generate: `openssl rand -base64 32`). +2. Confirm Pokémon TCG API key is present if set discovery uses authenticated endpoints. +3. After first cron run, spot-check Vercel function logs + `cards` row count for a + known recent set. +4. Optional: alert on cron failure (Vercel log drain / email) — not required for v1. + +## Relationship to scanner work + +| Scanner deliverable | How catalog sync helps | +| --- | --- | +| `card_submissions` queue (shipped) | Safety net when sync hasn't run yet | +| Disambiguation + "not listed" (in progress) | UX when catalog is stale | +| **This convoy** | Reduces stale-catalog frequency at the source | + +Queue **after** `redesign-scanner-flow`, `scanner-correctness-polish`, and +`rename-collections-vocabulary` unless operator reprioritizes. + +## Follow-up convoys (not v1) + +- **`reconcile-submissions-after-catalog-sync`** — when a set import lands, auto-match + pending `card_submissions` with matching `ocr_payload` set/name/number. +- **`lorcana-dynamic-set-discovery`** — replace hardcoded `setCodeMap` in + `import-lorcana.js`. +- **`catalog-sync-runs-table`** — migration for audit trail if console logs prove + insufficient. + +## Test plan + +- Unit: set-diff logic (mock DB rows vs mock API set list). +- Integration (staging): cron endpoint with `CRON_SECRET` imports one known small set; + second run skips all (idempotent). +- Manual: verify admin `/admin/card-import` still works after lib extraction. diff --git a/.convoys/ship-readiness.md b/.convoys/ship-readiness.md index 2693fbe..d0640d3 100644 --- a/.convoys/ship-readiness.md +++ b/.convoys/ship-readiness.md @@ -531,6 +531,23 @@ Six convoys authored from the scanner audit portfolio plan. Dependency order: Do not fold into the six scanner convoys above; queue as its own architect-led migration convoy after the scan pipeline stabilizes. +### Catalog freshness (deferred — post-scanner) + +- **`catalog-sync-vercel-cron`** (priority: P2 infra / data hygiene; + **deferred until scanner pipeline convoys finish**). Operator decision + 2026-05-27: use **Vercel Cron** (not GitHub Actions) for weekly delta sync of + new MTG + Pokémon sets. Motivation: Perfect Order Seel scan showed the catalog + has no row for unreleased/unimported sets; `card_submissions` is the safety net + but does not replace keeping `cards` current. v1 scope: extract shared import + logic from `import-mtg` / `import-pokemon`, discover missing sets via Scryfall + + Pokémon TCG API `/sets`, protected `/api/cron/sync-catalog` with `CRON_SECRET`, + `vercel.json` weekly schedule, paced imports respecting upstream rate limits. + Lorcana auto-discovery deferred (hardcoded `setCodeMap` today). **Do not start + until** `redesign-scanner-flow`, `scanner-correctness-polish`, and in-flight + scanner fixes (catalog-gap disambiguation, foil vision) are merged — operator + explicitly requested finishing scanner work first. Convoy: + `.convoys/catalog-sync-vercel-cron.md`. + ## Self-analytics After each convoy, `scripts/log-convoy-event.sh` emits a record to `.convoys/.metrics.jsonl` (gitignored). After 3-5 convoys, run the upstream `agent-pipeline/analytics/` aggregator to see where token spend goes — that data feeds whether to add or remove rules. diff --git a/components/CameraScanner.js b/components/CameraScanner.js index caf62b0..bdabc0b 100644 --- a/components/CameraScanner.js +++ b/components/CameraScanner.js @@ -4,6 +4,8 @@ export default function CameraScanner({ onCardScanned, onError }) { const [isStreaming, setIsStreaming] = useState(false); const [isDetecting, setIsDetecting] = useState(false); const [disambiguation, setDisambiguation] = useState(null); + const [scanNotice, setScanNotice] = useState(null); + const [submittingReview, setSubmittingReview] = useState(false); const videoRef = useRef(null); const canvasRef = useRef(null); @@ -19,6 +21,7 @@ export default function CameraScanner({ onCardScanned, onError }) { const visionCooldownUntilRef = useRef(0); const activeVerificationRef = useRef(0); const lastErrorAtRef = useRef(0); + const disambiguationRefineRef = useRef(null); // Mana symbol settings const [manaSymbolSettings, setManaSymbolSettings] = useState({ useSVG: false }); @@ -276,6 +279,72 @@ export default function CameraScanner({ onCardScanned, onError }) { const { cardTracker, imageData, ocrMeta } = disambiguation; emitScannedCard(cardTracker, imageData, candidate, ocrMeta); setDisambiguation(null); + disambiguationRefineRef.current = null; + }; + + const showScanNotice = (message) => { + setScanNotice(message); + setTimeout(() => setScanNotice(null), 8000); + }; + + const handleReviewSubmitted = (cardTracker, message) => { + if (cardTracker) cardTracker.status = 'confirmed'; + setDisambiguation(null); + disambiguationRefineRef.current = null; + showScanNotice(message); + }; + + const handleNotInCatalog = async () => { + if (!disambiguation || submittingReview) return; + setSubmittingReview(true); + + const { cardTracker, imageData, candidates, ocrMeta } = disambiguation; + const guessedName = + ocrMeta?.cardName || + ocrMeta?.query || + candidates?.[0]?.name || + null; + + try { + const response = await fetch('/api/scan/submit-for-review', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + Authorization: `Bearer ${localStorage.getItem('auth_token')}`, + }, + body: JSON.stringify({ + imageData, + name: guessedName, + candidateCardIds: (candidates || []).map((c) => c.id), + }), + }); + + if (response.status === 429) { + visionCooldownUntilRef.current = Date.now() + 60_000; + reportScannerError('Too many scan attempts. Please wait a moment and try again.'); + return; + } + + if (!response.ok) { + const errBody = await response.json().catch(() => ({})); + throw new Error(errBody.error || 'Failed to submit scan for review'); + } + + const result = await response.json(); + handleReviewSubmitted(cardTracker, result.message); + } catch (error) { + reportScannerError(error.message || 'Failed to submit scan for review'); + } finally { + setSubmittingReview(false); + } + }; + + const candidateMatchesSetHint = (candidate, setName, setCode) => { + if (!setName && !setCode) return true; + const hint = (setName || setCode || '').toLowerCase(); + const setNameLower = (candidate.set_name || '').toLowerCase(); + const setCodeLower = (candidate.set_code || '').toLowerCase(); + return setNameLower.includes(hint) || hint.includes(setNameLower) || setCodeLower === hint; }; const processIdentifyResponse = async (cardTracker, imageData, result) => { @@ -311,9 +380,15 @@ export default function CameraScanner({ onCardScanned, onError }) { return; } - if (result.needsReview || result.needsUserInput) { + if (result.needsReview) { + cardTracker.status = 'confirmed'; + showScanNotice(result.message || 'Scan saved for admin review.'); + return; + } + + if (result.needsUserInput) { cardTracker.status = 'negative'; - reportScannerError(result.message || 'Could not identify card — saved for review or retry.'); + reportScannerError(result.message || 'Could not identify card — try again or submit for review.'); return; } @@ -322,12 +397,133 @@ export default function CameraScanner({ onCardScanned, onError }) { }; const reportScannerError = (message) => { + if (disambiguation) return; const now = Date.now(); if (now - lastErrorAtRef.current < 4000) return; lastErrorAtRef.current = now; onError?.(message); }; + useEffect(() => { + if (!disambiguation?.imageData) return; + if (Date.now() < visionCooldownUntilRef.current) return; + + const refineKey = disambiguation.cardTracker?.id ?? 'modal'; + if (disambiguationRefineRef.current === refineKey) return; + disambiguationRefineRef.current = refineKey; + + let cancelled = false; + + (async () => { + try { + const response = await fetch('/api/scan/identify', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + Authorization: `Bearer ${localStorage.getItem('auth_token')}`, + }, + body: JSON.stringify({ imageData: disambiguation.imageData }), + }); + + if (cancelled) return; + + if (response.status === 429) { + visionCooldownUntilRef.current = Date.now() + 60_000; + return; + } + + if (!response.ok) return; + + const result = await response.json(); + if (cancelled) return; + + if (result.needsReview) { + handleReviewSubmitted(disambiguation.cardTracker, result.message); + return; + } + + if (result.card) { + emitScannedCard( + disambiguation.cardTracker, + disambiguation.imageData, + result.card, + { + confidence: result.ocr?.confidence, + rawText: result.ocr?.rawText, + abilities: result.card?.ocr?.abilities || [], + } + ); + setDisambiguation(null); + disambiguationRefineRef.current = null; + return; + } + + const setHint = result.ocr?.setName || result.ocr?.setCode; + if (result.matches?.length && setHint) { + const filtered = disambiguation.candidates.filter((candidate) => + candidateMatchesSetHint(candidate, result.ocr.setName, result.ocr.setCode) + ); + + if (filtered.length === 0 && setHint) { + try { + const submitRes = await fetch('/api/scan/submit-for-review', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + Authorization: `Bearer ${localStorage.getItem('auth_token')}`, + }, + body: JSON.stringify({ + imageData: disambiguation.imageData, + name: result.ocr?.cardName || disambiguation.candidates[0]?.name, + candidateCardIds: disambiguation.candidates.map((c) => c.id), + }), + }); + if (submitRes.ok) { + const submitResult = await submitRes.json(); + handleReviewSubmitted(disambiguation.cardTracker, submitResult.message); + } + } catch { + setDisambiguation((current) => + current + ? { + ...current, + visionHint: setHint, + message: `No "${setHint}" printing in our catalog. Tap "My card isn't listed" to save for admin review.`, + } + : current + ); + } + return; + } + + if (filtered.length === 1) { + handleDisambiguationPick(filtered[0]); + return; + } + + if (filtered.length > 1 && filtered.length < disambiguation.candidates.length) { + setDisambiguation((current) => + current + ? { + ...current, + candidates: filtered, + message: `Narrowed to ${filtered.length} printings matching "${setHint}".`, + visionHint: setHint, + } + : current + ); + } + } + } catch (error) { + console.warn('Disambiguation vision refine failed:', error); + } + })(); + + return () => { + cancelled = true; + }; + }, [disambiguation?.cardTracker?.id, disambiguation?.imageData]); + // Server-side card identification const verifyCardShape = async (cardTracker) => { if (!videoRef.current || !canvasRef.current || cardTracker.status !== 'detecting') return; @@ -568,6 +764,19 @@ export default function CameraScanner({ onCardScanned, onError }) { return (
+ {scanNotice && ( +
+ {scanNotice} +
+ )} {/* Camera Feed Container */}
{disambiguation.message || 'Multiple matches found. Select the correct printing.'}

+ {disambiguation.visionHint && ( +

+ Vision detected set: {disambiguation.visionHint} +

+ )}
{disambiguation.candidates.map((candidate) => ( +