deckhearth/.convoys/scan-visual-catalog-search.md

---
name: scan-visual-catalog-search
classification: feature
success_metric: |
  After catalog embeddings exist and the new kNN path is on, ≥50% of
  legitimate card scans (scan_attempts excluding not_a_card) auto-match
  a printing with no picker — the original add-real-ocr-layer 70%
  target, measured honestly. Gemini L2 share of attempts falls vs the
  Phase 1 post-ship baseline.
skip:
  - ia
  - ui-design
  - visual
  - a11y
  - design
  - flag
status: shipped
created: 2026-08-14
depends_on:
  - improve-scan-card-detection
umbrella: scanner-identify-upgrade
model_policy:
  default_session: auto
  roles:
    role-conductor: composer-2.5-fast
    role-architect: composer-2.5
    role-ia-architect: composer-2.5-fast
    role-ux-reviewer: composer-2.5-fast
    role-ui-designer: composer-2.5-fast
    role-implementer: composer-2.5-fast
    role-reviewer: cursor-grok-4.5-high
    role-security-auditor: gpt-5.6-terra-medium
    role-design-system-auditor: cursor-grok-4.5-high
    role-a11y-auditor: cursor-grok-4.5-high
    role-doc-writer: auto
  escalate_to: claude-sonnet-5-thinking-medium
  escalate_to_premium: claude-4.6-opus-high-thinking
  never_premium:
    - role-reviewer
    - role-security-auditor
    - role-design-system-auditor
    - role-a11y-auditor
    - role-ui-designer
    - role-doc-writer
---

# Convoy: scan-visual-catalog-search

Phase 3 of `scanner-identify-upgrade`. Identify cards by **what they
look like**, not by reading the name. This is how Manabox / Delver-style
scanners get printing-accurate matches on foil and alt-art.

Blocked on `improve-scan-card-detection`: embeddings of unwarped
phone crops will not match catalog `image_url`s reliably.

## Why

Name-only `pg_trgm` cannot distinguish printings. Tesseract fails on
foil and stylized type even when it emits 8–20 characters (76.7% L1
escalate with non-empty text). Gemini can read a name but still
returns a picker when set/number are missing, and costs 5/min.

The catalog already stores `cards.image_url` for every imported
printing. Precompute an embedding per row; at scan time embed the
warped crop and take top-k cosine. OCR / Gemini become hints and
unknown-card fallback.

## Scope

### In scope

- **Migration:** `pgvector` (confirm Neon availability) +
  `cards.embedding` (or a side table `card_embeddings`) + ANN index.
  Update `docs/SCHEMA_MAP.md`.
- **Offline embed job:** new dated script or `npm run` task that
  reads `image_url`, writes vectors. Idempotent. Rate-limit the
  embedding provider. Do **not** edit historical `scripts/add-*.js`.
- **Architect picks the embedder** (one):
  1. Gateway embedding model (same `AI_GATEWAY_API_KEY`, server-only).
  2. In-browser MobileCLIP-S2 / SigLIP ONNX for the *query* crop, with
     catalog vectors baked or fetched — only if weight license + size
     are acceptable.
- **Identify path:** new Layer-0 (or replace L1) — kNN then
  auto-match / disambiguate / escalate to existing L2. Log
  `scan_attempts.layer = 0` (or Architect-ratified value).
- **Similarity thresholds** analogous to 0.85 / 0.60, tuned on a
  held-out set of scan captures if any exist in Blob.
- Auth + rate-limit on any new route. No client-side API keys
  (`forbidden-client-side-llm-keys` must stay green).

### Out of scope

- Replacing Gemini entirely on day one — keep L2 for catalog misses
  and low similarity.
- Training a custom card CNN.
- Python GPU service.
- Changing scanner chrome / cart.
- Auto-approving the 27 pending `card_submissions`.

## Roles invoked

1. `role-architect` — embedder, schema, layer numbering, thresholds,
   brief split (migration / backfill / route). Security-sensitive:
   escalate to Sonnet if schema + new route land together.
2. `role-implementer`.
3. Audit: `role-reviewer` + `role-security-auditor` (required).

## Todos

- [x] Architect: confirm `pgvector` on prod Neon tier
- [x] Brief 1 — migration + SCHEMA_MAP
- [x] Brief 2 — catalog backfill job (idempotent)
- [x] Brief 3 — identify kNN route + client escalate order
- [ ] Threshold bake-off on real crops
- [ ] Re-measure auto-match % excluding `not_a_card`
- [ ] Operator: `AI_GATEWAY_API_KEY` + `npm run backfill-embeddings` on Neon (migrate applied 2026-08-15)

## Post-ship (2026-08-15)

**CI fixes:** PR #162 — `lib/card-embed.js` allowlist + pgvector migration
graceful skip on non-superuser homelab CI.

**Neon operator:** `1782000000001_add-card-embeddings` applied manually
(pgvector + columns + index + pgmigrations row). Full backfill pending
`AI_GATEWAY_API_KEY` in operator env.

**scan_attempts snapshot** (90d, n=250, pre-backfill / no L0 traffic yet):

| Signal | Value | Baseline |
| --- | --- | --- |
| L1 escalate | 78.4% (87/111) | 76.7% |
| L1 matched | 5.4% (6/111) | 5.8% |
| L2 not_a_card | 43.9% (61/139) | 44.6% |
| End-to-end auto-match | 10.0% (25/250) | 10.3% |

Re-measure after backfill + preview scanning for L0 `layer=0` rows.

## Likely file ownership

| Area | Files |
| --- | --- |
| Schema | `migrations/*_card-embeddings.js`, `docs/SCHEMA_MAP.md` |
| Backfill | new `scripts/` job (dated) or `lib/card-embed-backfill.js` |
| Query | new `lib/card-visual-match.js`, `pages/api/scan/identify-by-image.js` **or** fold into existing identify |
| Client | `lib/scanner-card-identify.js` (`tryLayer1TextIdentify` sibling) |

Do not rewrite `lib/scanner-card-detection.js` here.

## Multitask dispatch

Brief 1 first. Brief 2 after 1. Brief 3 after 1 (can overlap 2 if the
route degrades to escalate-when-empty-index).

```
/multitask role-reviewer + role-security-auditor
```

Group id: `audit-scan-visual-catalog-search-<pr>`.

## CI impact

| Workflow / job | Behavior |
| --- | --- |
| `schema-map-fresh` | **Fires** — migration + SCHEMA_MAP |
| `ci.yml` migrate | Must apply `pgvector` on CT 102 CI Postgres — Architect must verify the extension is available there or gate the migration |
| `forbidden-client-side-llm-keys` | Blocking |

## Operator action

- Confirm Neon `pgvector` (or Neon’s equivalent) on the prod project.
- Budget: one embedding per catalog image, plus one per live scan if
  the query embed is server-side. Architect publishes a cost note
  before Brief 2 runs against prod images.
- No new browser secrets.

## Conductor notes

This is the accuracy leap. Do not start it to "try CLIP" before Phase 2
crops are rectified — that wastes the backfill. If Architect finds
`pgvector` unavailable on CI Postgres, stop and write a fallback
(external index vs skip-CI-extension plan) rather than shipping an
untestable migration.

## UX

No new screens. Scan flow stays L0 → L1 → L2 with the same
disambiguation picker and error toasts. When the catalog index is empty
(backfill not run), L0 escalates silently with no embed cost.

Gallery uploads now try visual match before OCR.

## Architecture

### Decision D1 — Gateway multimodal embedder (`cohere/embed-v4.0`)

Server-only via `AI_GATEWAY_API_KEY`. 1024-dim vectors in
`cards.embedding`. Env: `SCAN_EMBED_MODEL`, `SCAN_EMBED_DIMENSION`.

### Decision D2 — Layer numbering

| Layer | Path |
| --- | --- |
| 0 | `POST /api/scan/identify-by-image` |
| 1 | Tesseract + `identify-by-text` |
| 2 | Gemini + `scan/identify` |

### Decision D3 — Thresholds

Match ≥ **0.82** (0.06 gap). Disambiguation ≥ **0.58**.

### Decision D4 — Rate limit

`checkScanRateLimit` on identify-by-image. L0 429 falls through to L1
(not a hard stop).

Audit group id: `audit-scan-visual-catalog-search-<pr>`.