deckhearth/.convoys/scan-visual-catalog-search.md

name	classification	success_metric	skip	status	created	depends_on	umbrella	model_policy
scan-visual-catalog-search	feature	After catalog embeddings exist and the new kNN path is on, ≥50% of legitimate card scans (scan_attempts excluding not_a_card) auto-match a printing with no picker — the original add-real-ocr-layer 70% target, measured honestly. Gemini L2 share of attempts falls vs the Phase 1 post-ship baseline.	ia ui-design visual a11y design flag	open	2026-08-14	improve-scan-card-detection	scanner-identify-upgrade	default_session roles escalate_to escalate_to_premium never_premium auto role-conductor role-architect role-ia-architect role-ux-reviewer role-ui-designer role-implementer role-reviewer role-security-auditor role-design-system-auditor role-a11y-auditor role-doc-writer composer-2.5-fast composer-2.5 composer-2.5-fast composer-2.5-fast composer-2.5-fast composer-2.5-fast cursor-grok-4.5-high gpt-5.6-terra-medium cursor-grok-4.5-high cursor-grok-4.5-high auto claude-sonnet-5-thinking-medium claude-4.6-opus-high-thinking role-reviewer role-security-auditor role-design-system-auditor role-a11y-auditor role-ui-designer role-doc-writer

Convoy: scan-visual-catalog-search

Phase 3 of scanner-identify-upgrade. Identify cards by what they look like, not by reading the name. This is how Manabox / Delver-style scanners get printing-accurate matches on foil and alt-art.

Blocked on improve-scan-card-detection: embeddings of unwarped phone crops will not match catalog image_urls reliably.

Why

Name-only pg_trgm cannot distinguish printings. Tesseract fails on foil and stylized type even when it emits 8–20 characters (76.7% L1 escalate with non-empty text). Gemini can read a name but still returns a picker when set/number are missing, and costs 5/min.

The catalog already stores cards.image_url for every imported printing. Precompute an embedding per row; at scan time embed the warped crop and take top-k cosine. OCR / Gemini become hints and unknown-card fallback.

Scope

In scope

• Migration: pgvector (confirm Neon availability) + cards.embedding (or a side table card_embeddings) + ANN index. Update docs/SCHEMA_MAP.md.
• Offline embed job: new dated script or npm run task that reads image_url, writes vectors. Idempotent. Rate-limit the embedding provider. Do not edit historical scripts/add-*.js.
• Architect picks the embedder (one):
  1. Gateway embedding model (same AI_GATEWAY_API_KEY, server-only).
  2. In-browser MobileCLIP-S2 / SigLIP ONNX for the query crop, with catalog vectors baked or fetched — only if weight license + size are acceptable.
• Identify path: new Layer-0 (or replace L1) — kNN then auto-match / disambiguate / escalate to existing L2. Log scan_attempts.layer = 0 (or Architect-ratified value).
• Similarity thresholds analogous to 0.85 / 0.60, tuned on a held-out set of scan captures if any exist in Blob.
• Auth + rate-limit on any new route. No client-side API keys (forbidden-client-side-llm-keys must stay green).

Out of scope

• Replacing Gemini entirely on day one — keep L2 for catalog misses and low similarity.
• Training a custom card CNN.
• Python GPU service.
• Changing scanner chrome / cart.
• Auto-approving the 27 pending card_submissions.

Roles invoked

1. role-architect — embedder, schema, layer numbering, thresholds, brief split (migration / backfill / route). Security-sensitive: escalate to Sonnet if schema + new route land together.
2. role-implementer.
3. Audit: role-reviewer + role-security-auditor (required).

Todos

• Architect: confirm pgvector on prod Neon tier
• Brief 1 — migration + SCHEMA_MAP
• Brief 2 — catalog backfill job (idempotent)
• Brief 3 — identify kNN route + client escalate order
• Threshold bake-off on real crops
• Re-measure auto-match % excluding not_a_card

Likely file ownership

Area	Files
Schema	migrations/*_card-embeddings.js, docs/SCHEMA_MAP.md
Backfill	new scripts/ job (dated) or lib/card-embed-backfill.js
Query	new lib/card-visual-match.js, pages/api/scan/identify-by-image.js or fold into existing identify
Client	lib/scanner-card-identify.js (tryLayer1TextIdentify sibling)

Do not rewrite lib/scanner-card-detection.js here.

Multitask dispatch

Brief 1 first. Brief 2 after 1. Brief 3 after 1 (can overlap 2 if the route degrades to escalate-when-empty-index).

/multitask role-reviewer + role-security-auditor

Group id: audit-scan-visual-catalog-search-<pr>.

CI impact

Workflow / job	Behavior
schema-map-fresh	Fires — migration + SCHEMA_MAP
ci.yml migrate	Must apply pgvector on CT 102 CI Postgres — Architect must verify the extension is available there or gate the migration
forbidden-client-side-llm-keys	Blocking

Operator action

• Confirm Neon pgvector (or Neon’s equivalent) on the prod project.
• Budget: one embedding per catalog image, plus one per live scan if the query embed is server-side. Architect publishes a cost note before Brief 2 runs against prod images.
• No new browser secrets.

Conductor notes

This is the accuracy leap. Do not start it to "try CLIP" before Phase 2 crops are rectified — that wastes the backfill. If Architect finds pgvector unavailable on CI Postgres, stop and write a fallback (external index vs skip-CI-extension plan) rather than shipping an untestable migration.