c6a0225e54
Record Aug 15 measurement showing Phases 1–2 missed success targets, Phase 3 code is live with 15% embedding backfill and zero L0 traffic yet. Co-authored-by: Cursor <cursoragent@cursor.com>
233 lines
8.1 KiB
Markdown
233 lines
8.1 KiB
Markdown
---
|
||
name: scan-visual-catalog-search
|
||
classification: feature
|
||
success_metric: |
|
||
After catalog embeddings exist and the new kNN path is on, ≥50% of
|
||
legitimate card scans (scan_attempts excluding not_a_card) auto-match
|
||
a printing with no picker — the original add-real-ocr-layer 70%
|
||
target, measured honestly. Gemini L2 share of attempts falls vs the
|
||
Phase 1 post-ship baseline.
|
||
skip:
|
||
- ia
|
||
- ui-design
|
||
- visual
|
||
- a11y
|
||
- design
|
||
- flag
|
||
status: shipped
|
||
created: 2026-08-14
|
||
depends_on:
|
||
- improve-scan-card-detection
|
||
umbrella: scanner-identify-upgrade
|
||
model_policy:
|
||
default_session: auto
|
||
roles:
|
||
role-conductor: composer-2.5-fast
|
||
role-architect: composer-2.5
|
||
role-ia-architect: composer-2.5-fast
|
||
role-ux-reviewer: composer-2.5-fast
|
||
role-ui-designer: composer-2.5-fast
|
||
role-implementer: composer-2.5-fast
|
||
role-reviewer: cursor-grok-4.5-high
|
||
role-security-auditor: gpt-5.6-terra-medium
|
||
role-design-system-auditor: cursor-grok-4.5-high
|
||
role-a11y-auditor: cursor-grok-4.5-high
|
||
role-doc-writer: auto
|
||
escalate_to: claude-sonnet-5-thinking-medium
|
||
escalate_to_premium: claude-4.6-opus-high-thinking
|
||
never_premium:
|
||
- role-reviewer
|
||
- role-security-auditor
|
||
- role-design-system-auditor
|
||
- role-a11y-auditor
|
||
- role-ui-designer
|
||
- role-doc-writer
|
||
---
|
||
|
||
# Convoy: scan-visual-catalog-search
|
||
|
||
Phase 3 of `scanner-identify-upgrade`. Identify cards by **what they
|
||
look like**, not by reading the name. This is how Manabox / Delver-style
|
||
scanners get printing-accurate matches on foil and alt-art.
|
||
|
||
Blocked on `improve-scan-card-detection`: embeddings of unwarped
|
||
phone crops will not match catalog `image_url`s reliably.
|
||
|
||
## Why
|
||
|
||
Name-only `pg_trgm` cannot distinguish printings. Tesseract fails on
|
||
foil and stylized type even when it emits 8–20 characters (76.7% L1
|
||
escalate with non-empty text). Gemini can read a name but still
|
||
returns a picker when set/number are missing, and costs 5/min.
|
||
|
||
The catalog already stores `cards.image_url` for every imported
|
||
printing. Precompute an embedding per row; at scan time embed the
|
||
warped crop and take top-k cosine. OCR / Gemini become hints and
|
||
unknown-card fallback.
|
||
|
||
## Scope
|
||
|
||
### In scope
|
||
|
||
- **Migration:** `pgvector` (confirm Neon availability) +
|
||
`cards.embedding` (or a side table `card_embeddings`) + ANN index.
|
||
Update `docs/SCHEMA_MAP.md`.
|
||
- **Offline embed job:** new dated script or `npm run` task that
|
||
reads `image_url`, writes vectors. Idempotent. Rate-limit the
|
||
embedding provider. Do **not** edit historical `scripts/add-*.js`.
|
||
- **Architect picks the embedder** (one):
|
||
1. Gateway embedding model (same `AI_GATEWAY_API_KEY`, server-only).
|
||
2. In-browser MobileCLIP-S2 / SigLIP ONNX for the *query* crop, with
|
||
catalog vectors baked or fetched — only if weight license + size
|
||
are acceptable.
|
||
- **Identify path:** new Layer-0 (or replace L1) — kNN then
|
||
auto-match / disambiguate / escalate to existing L2. Log
|
||
`scan_attempts.layer = 0` (or Architect-ratified value).
|
||
- **Similarity thresholds** analogous to 0.85 / 0.60, tuned on a
|
||
held-out set of scan captures if any exist in Blob.
|
||
- Auth + rate-limit on any new route. No client-side API keys
|
||
(`forbidden-client-side-llm-keys` must stay green).
|
||
|
||
### Out of scope
|
||
|
||
- Replacing Gemini entirely on day one — keep L2 for catalog misses
|
||
and low similarity.
|
||
- Training a custom card CNN.
|
||
- Python GPU service.
|
||
- Changing scanner chrome / cart.
|
||
- Auto-approving the 27 pending `card_submissions`.
|
||
|
||
## Roles invoked
|
||
|
||
1. `role-architect` — embedder, schema, layer numbering, thresholds,
|
||
brief split (migration / backfill / route). Security-sensitive:
|
||
escalate to Sonnet if schema + new route land together.
|
||
2. `role-implementer`.
|
||
3. Audit: `role-reviewer` + `role-security-auditor` (required).
|
||
|
||
## Todos
|
||
|
||
- [x] Architect: confirm `pgvector` on prod Neon tier
|
||
- [x] Brief 1 — migration + SCHEMA_MAP
|
||
- [x] Brief 2 — catalog backfill job (idempotent)
|
||
- [x] Brief 3 — identify kNN route + client escalate order
|
||
- [x] Threshold bake-off on real crops (initial: 0.82 match / 0.58 disambig — tune post-backfill)
|
||
- [ ] Re-measure auto-match % excluding `not_a_card` (blocked: L0 traffic + backfill)
|
||
- [ ] Operator: finish `npm run backfill-embeddings` — **14.9%** (9,865 / 66,211) as of 2026-08-15
|
||
|
||
## Post-ship (2026-08-15)
|
||
|
||
**Merged:** PR #160 (Layer-0 visual catalog search), PR #162 (CI gates).
|
||
|
||
**Pipeline roles (kickoff complete):**
|
||
|
||
| Role | Outcome |
|
||
| --- | --- |
|
||
| Conductor | Convoy ratified; skip ia/ui-design/visual/a11y/design/flag |
|
||
| UX reviewer | No new screens; L0→L1→L2 order; silent escalate when index empty |
|
||
| Architect | D1–D4 decisions + 3 briefs (`.convoys/scan-visual-catalog-search/brief-*`) |
|
||
| Implementer | Briefs 1–3 shipped in #160 |
|
||
| Reviewer + security-auditor | #160 audit fan-out; #162 CI fix |
|
||
| Doc-writer | SCHEMA_MAP layer 0 documented |
|
||
|
||
**CI fixes (#162):** `lib/card-embed.js` allowlist + pgvector migration graceful
|
||
skip on non-superuser homelab CI.
|
||
|
||
**Neon:** pgvector v0.8.0; `cards.embedding vector(1024)` + HNSW index applied.
|
||
Backfill **in progress** (9,865 / 66,211 = 14.9%). **0** `scan_attempts` with
|
||
`layer = 0` yet — L0 path live but index too sparse / no post-#160 scans logged.
|
||
|
||
**scan_attempts snapshot** (all-time n=250):
|
||
|
||
| Signal | Value | Baseline |
|
||
| --- | --- | --- |
|
||
| L1 escalate | 78.4% (87/111) | 76.7% |
|
||
| L1 matched | 5.4% (6/111) | 5.8% |
|
||
| L2 not_a_card | 43.9% (61/139) | 44.6% |
|
||
| End-to-end auto-match | 13.2% excl. not_a_card | 13.5% |
|
||
| L0 attempts | 0 | — |
|
||
|
||
**Aug 15 session (n=26):** See umbrella `.convoys/scanner-identify-upgrade.md`
|
||
§ Post-ship telemetry.
|
||
|
||
Re-measure after backfill completes and preview scanning generates L0 rows.
|
||
|
||
## Likely file ownership
|
||
|
||
| Area | Files |
|
||
| --- | --- |
|
||
| Schema | `migrations/*_card-embeddings.js`, `docs/SCHEMA_MAP.md` |
|
||
| Backfill | new `scripts/` job (dated) or `lib/card-embed-backfill.js` |
|
||
| Query | new `lib/card-visual-match.js`, `pages/api/scan/identify-by-image.js` **or** fold into existing identify |
|
||
| Client | `lib/scanner-card-identify.js` (`tryLayer1TextIdentify` sibling) |
|
||
|
||
Do not rewrite `lib/scanner-card-detection.js` here.
|
||
|
||
## Multitask dispatch
|
||
|
||
Brief 1 first. Brief 2 after 1. Brief 3 after 1 (can overlap 2 if the
|
||
route degrades to escalate-when-empty-index).
|
||
|
||
```
|
||
/multitask role-reviewer + role-security-auditor
|
||
```
|
||
|
||
Group id: `audit-scan-visual-catalog-search-<pr>`.
|
||
|
||
## CI impact
|
||
|
||
| Workflow / job | Behavior |
|
||
| --- | --- |
|
||
| `schema-map-fresh` | **Fires** — migration + SCHEMA_MAP |
|
||
| `ci.yml` migrate | Must apply `pgvector` on CT 102 CI Postgres — Architect must verify the extension is available there or gate the migration |
|
||
| `forbidden-client-side-llm-keys` | Blocking |
|
||
|
||
## Operator action
|
||
|
||
- Confirm Neon `pgvector` (or Neon’s equivalent) on the prod project.
|
||
- Budget: one embedding per catalog image, plus one per live scan if
|
||
the query embed is server-side. Architect publishes a cost note
|
||
before Brief 2 runs against prod images.
|
||
- No new browser secrets.
|
||
|
||
## Conductor notes
|
||
|
||
This is the accuracy leap. Do not start it to "try CLIP" before Phase 2
|
||
crops are rectified — that wastes the backfill. If Architect finds
|
||
`pgvector` unavailable on CI Postgres, stop and write a fallback
|
||
(external index vs skip-CI-extension plan) rather than shipping an
|
||
untestable migration.
|
||
|
||
## UX
|
||
|
||
No new screens. Scan flow stays L0 → L1 → L2 with the same
|
||
disambiguation picker and error toasts. When the catalog index is empty
|
||
(backfill not run), L0 escalates silently with no embed cost.
|
||
|
||
Gallery uploads now try visual match before OCR.
|
||
|
||
## Architecture
|
||
|
||
### Decision D1 — Gateway multimodal embedder (`cohere/embed-v4.0`)
|
||
|
||
Server-only via `AI_GATEWAY_API_KEY`. 1024-dim vectors in
|
||
`cards.embedding`. Env: `SCAN_EMBED_MODEL`, `SCAN_EMBED_DIMENSION`.
|
||
|
||
### Decision D2 — Layer numbering
|
||
|
||
| Layer | Path |
|
||
| --- | --- |
|
||
| 0 | `POST /api/scan/identify-by-image` |
|
||
| 1 | Tesseract + `identify-by-text` |
|
||
| 2 | Gemini + `scan/identify` |
|
||
|
||
### Decision D3 — Thresholds
|
||
|
||
Match ≥ **0.82** (0.06 gap). Disambiguation ≥ **0.58**.
|
||
|
||
### Decision D4 — Rate limit
|
||
|
||
`checkScanRateLimit` on identify-by-image. L0 429 falls through to L1
|
||
(not a hard stop).
|
||
|
||
Audit group id: `audit-scan-visual-catalog-search-<pr>`.
|