c6a0225e54
Record Aug 15 measurement showing Phases 1–2 missed success targets, Phase 3 code is live with 15% embedding backfill and zero L0 traffic yet. Co-authored-by: Cursor <cursoragent@cursor.com>
265 lines
10 KiB
Markdown
265 lines
10 KiB
Markdown
---
|
||
name: scanner-identify-upgrade
|
||
classification: feature
|
||
success_metric: |
|
||
Of legitimate card scans (excluding not_a_card), ≥50% auto-match a
|
||
catalog printing with no picker; Layer-1 escalate rate falls from 76.7%
|
||
to ≤40%; median /api/scan/identify latency stays near today's 1.7s p50
|
||
or improves. Measured on scan_attempts after Phase 1 ships.
|
||
skip: []
|
||
status: open
|
||
created: 2026-08-14
|
||
depends_on:
|
||
- add-real-ocr-layer
|
||
- server-side-scan-pipeline
|
||
model_policy:
|
||
default_session: auto
|
||
roles:
|
||
role-conductor: composer-2.5-fast
|
||
role-architect: composer-2.5
|
||
role-ia-architect: composer-2.5-fast
|
||
role-ux-reviewer: composer-2.5-fast
|
||
role-ui-designer: composer-2.5-fast
|
||
role-implementer: composer-2.5-fast
|
||
role-reviewer: cursor-grok-4.5-high
|
||
role-security-auditor: gpt-5.6-terra-medium
|
||
role-design-system-auditor: cursor-grok-4.5-high
|
||
role-a11y-auditor: cursor-grok-4.5-high
|
||
role-doc-writer: auto
|
||
escalate_to: claude-sonnet-5-thinking-medium
|
||
escalate_to_premium: claude-4.6-opus-high-thinking
|
||
never_premium:
|
||
- role-reviewer
|
||
- role-security-auditor
|
||
- role-design-system-auditor
|
||
- role-a11y-auditor
|
||
- role-ui-designer
|
||
- role-doc-writer
|
||
---
|
||
|
||
# Scanner identify upgrade — epic
|
||
|
||
Umbrella for making the camera scanner **faster and more accurate**
|
||
without replacing the two-layer shape (cheap local path, then
|
||
server-owned Gemini). Planning-only in this file. Each numbered
|
||
sub-convoy is its own gated PR stream.
|
||
|
||
Worktree: `tcg-vault-worktrees/scanner-identify-upgrade` on
|
||
`convoy/scanner-identify-upgrade` (branched from `main` @ `c6c1364`).
|
||
Do **not** land these docs on `feat/scanner-mobile-checkout` — that
|
||
convoy owns scanner chrome/cart and explicitly leaves identify out of
|
||
scope.
|
||
|
||
## Why
|
||
|
||
`add-real-ocr-layer` (PR #38) targeted ≥70% of scans resolving at
|
||
Layer-1 (Tesseract + `pg_trgm`) with zero Gemini calls. Live
|
||
`scan_attempts` from the local Neon (2026-05-27 → 2026-08-12, n=224,
|
||
1 user) shows that target was missed by a wide margin:
|
||
|
||
| Signal | Value | Target / note |
|
||
| --- | --- | --- |
|
||
| L1 auto-match (`result_kind=matched`) | **5.8% of L1** / 2.7% of all | ≥70% L1 resolve |
|
||
| L1 escalate | **76.7% of L1** (79/103) | Should be the minority path |
|
||
| L1 disambiguation | 17.5% of L1 (18/103) | Name-only matcher cannot pick printings |
|
||
| End-to-end auto-match | **10.3%** (23/224) | User still picks or retries most cards |
|
||
| L2 `not_a_card` | **44.6% of L2** (54/121) | Detector fires on non-cards |
|
||
| L2 p50 / p90 latency | **1655 / 2349 ms** | Plus 3.5s hold-still *before* identify |
|
||
| L1 p50 latency | **158 ms** | Fast, but almost never uniquely matches |
|
||
| `card_submissions` | 27 pending, 0 reviewed | Catalog-gap queue is unread |
|
||
|
||
The six L1 "matches" include noisy Tesseract strips (`J ——`, `Rock
|
||
Jockey ©`) at confidence 36–68. L1 is a cheap filter that rarely
|
||
identifies a printing. Meanwhile every card waits
|
||
`DETECTION_START_DELAY_MS` (1s) + `MIN_FIRST_SEEN_MS_FOR_VERIFY` (2.5s)
|
||
before OCR starts, and L2 is capped at 5 Gemini calls / user / minute.
|
||
|
||
Users feel this as: hold still forever, then pick from a list, or get
|
||
"not a card" / "saved for review."
|
||
|
||
## Scope
|
||
|
||
### In scope (this epic)
|
||
|
||
1. **`tighten-scan-identify-hot-path`** — Phase 1. Cut the hold-still
|
||
gate, OCR collector number on L1, structured Gemini JSON, optional
|
||
`SCAN_VISION_MODEL` bump, stop the automatic second Gemini refine.
|
||
2. **`improve-scan-card-detection`** — Phase 2. Replace the 320×240
|
||
Sobel brute-force detector with a real card crop + perspective warp.
|
||
This was queued in `add-real-ocr-layer` and never opened.
|
||
3. **`scan-visual-catalog-search`** — Phase 3. Embed catalog
|
||
`image_url`s; nearest-neighbor the warped crop. Gemini becomes
|
||
fallback. This is the accuracy leap.
|
||
|
||
### Out of scope (this epic)
|
||
|
||
- Scanner chrome, cart, checkout (`scanner-mobile-checkout` /
|
||
`scanner-rebuild`). Identify libs only.
|
||
- Training a custom card CNN or standing up a Python/CUDA OCR service.
|
||
- Swapping Tesseract for EasyOCR / PaddleOCR as the *primary* identifier.
|
||
- Raising the L2 rate limit until Phase 1 hit-rate is re-measured.
|
||
- Admin review of the 27 pending `card_submissions` (ops, not this epic).
|
||
|
||
## Baseline (do not re-query to "start" Phase 1)
|
||
|
||
Pulled 2026-08-14 from local `.env.local` → Neon `scan_attempts`.
|
||
|
||
```
|
||
layer | result_kind | n | pct
|
||
1 | escalate | 79 | 35.3
|
||
1 | disambiguation | 18 | 8.0
|
||
1 | matched | 6 | 2.7
|
||
2 | not_a_card | 54 | 24.1
|
||
2 | submitted | 26 | 11.6
|
||
2 | disambiguation | 22 | 9.8
|
||
2 | matched | 17 | 7.6
|
||
2 | needs_input | 2 | 0.9
|
||
```
|
||
|
||
L1 escalate text-length buckets (all had ≥3 chars — Tesseract is
|
||
emitting text that `pg_trgm` cannot match): 3–7 chars n=28; 8–19 n=30;
|
||
20+ n=21.
|
||
|
||
Re-measure with the same grouping after each sub-convoy ships.
|
||
|
||
## Dependency graph
|
||
|
||
```
|
||
[baseline pulled 2026-08-14]
|
||
│
|
||
▼
|
||
┌───────────────────────────────────┐
|
||
│ 1. tighten-scan-identify-hot-path │
|
||
│ gates, collector #, schema JSON│
|
||
└───────────────┬───────────────────┘
|
||
│
|
||
▼
|
||
┌───────────────────────────────────┐
|
||
│ 2. improve-scan-card-detection │
|
||
│ detect + homography crop │
|
||
└───────────────┬───────────────────┘
|
||
│
|
||
▼
|
||
┌───────────────────────────────────┐
|
||
│ 3. scan-visual-catalog-search │
|
||
│ embeddings + pgvector kNN │
|
||
└───────────────────────────────────┘
|
||
```
|
||
|
||
**Strict-blockers:** #1 before #2 only if #2 would retune the same
|
||
stability constants — otherwise #1 (identify files) and #2 (detection
|
||
files) are file-disjoint and may run in parallel after Architect
|
||
confirms. #3 needs a stable crop (#2) to be worth the embedding job;
|
||
do not start #3 until #2 has a warped JPEG.
|
||
|
||
**Sibling:** `scanner-mobile-checkout` (other worktree / branch) must
|
||
not edit `lib/ocr-worker.js`, `lib/scan-vision.js`,
|
||
`lib/card-text-match.js`, `lib/scanner-card-identify.js`,
|
||
`lib/scanner-card-detection.js`, or `/api/scan/identify`.
|
||
|
||
## Roles invoked
|
||
|
||
Umbrella is planning-only. Each sub-convoy lists its own roles.
|
||
Default for #1: Architect → Implementer → audit fan-out (reviewer +
|
||
security-auditor). UX reviewer on #1 and #2. Skip IA and UI Designer
|
||
on all three (no new routes or visual language).
|
||
|
||
## Todos
|
||
|
||
- [x] Pull `scan_attempts` baseline (2026-08-14)
|
||
- [x] Open worktree `scanner-identify-upgrade` from `main`
|
||
- [x] Seed sub-convoys #1–#3
|
||
- [x] Architect: pick up `tighten-scan-identify-hot-path` first
|
||
- [x] Re-measure `scan_attempts` after all phases ship (2026-08-15 — see below)
|
||
- [x] Architect: `improve-scan-card-detection`
|
||
- [x] Architect: `scan-visual-catalog-search` after warped crops exist
|
||
- [ ] Operator: finish embedding backfill (`npm run backfill-embeddings`) — **14.9%** done (9,865 / 66,211)
|
||
- [ ] Re-measure after backfill + L0 traffic (`layer = 0` rows)
|
||
|
||
## Post-ship telemetry (2026-08-15)
|
||
|
||
Pulled from Neon `scan_attempts` + `cards.embedding` coverage. Single user
|
||
(`user_id = 3`); treat post-ship windows as directional, not statistically
|
||
significant.
|
||
|
||
### All-time (n=250, 2026-05-27 → 2026-08-15)
|
||
|
||
| Signal | Value | Baseline (2026-08-14) | Target |
|
||
| --- | --- | --- | --- |
|
||
| L1 auto-match | **5.4%** of L1 (6/111) | 5.8% | ≥25% |
|
||
| L1 escalate | **78.4%** of L1 (87/111) | 76.7% | ≤40% |
|
||
| L2 `not_a_card` | **43.9%** of L2 (61/139) | 44.6% | ≤15% |
|
||
| End-to-end auto-match | **13.2%** excl. not_a_card (25/189) | 13.5% | ≥50% |
|
||
| L1 p50 latency | **158 ms** | 158 ms | — |
|
||
| L2 p50 / p90 | **1642 / 2297 ms** | 1655 / 2349 ms | ≤1800 p50 |
|
||
| L0 attempts | **0** | — | (Phase 3 code live; index sparse) |
|
||
| `card_submissions` pending | **31** | 27 | ops |
|
||
|
||
### Aug 15 post-deploy session (n=26)
|
||
|
||
First scan burst after #156–#158 merged (~06:23–07:43 UTC). No L0 rows yet
|
||
(#160 landed same evening; backfill incomplete).
|
||
|
||
| Signal | Aug 15 (n=26) | Baseline |
|
||
| --- | --- | --- |
|
||
| L1 auto-match | **0%** (0/8 L1) | 5.8% |
|
||
| L1 escalate | **100%** (8/8 L1) | 76.7% |
|
||
| L2 `not_a_card` | **38.9%** (7/18 L2) | 44.6% |
|
||
| End-to-end auto-match | **10.5%** excl. not_a_card (2/19) | 13.5% |
|
||
|
||
**Qualitative notes:** L1 OCR strips post-warp are *shorter* garbage (`"7"`,
|
||
`"Tey"`, `"yr'"`) — collector-number path cannot help. L2 structured JSON
|
||
(`isCard: false`) works. Two L2 auto-matches (Hunger Hawk, Hog-Monkey).
|
||
Disambiguation still common (Seel printings).
|
||
|
||
### Phase verdict
|
||
|
||
| Phase | PR | Code | Success metrics |
|
||
| --- | --- | --- | --- |
|
||
| 1 `tighten-scan-identify-hot-path` | #156 | **Shipped** | ❌ L1 match/escalate targets missed |
|
||
| 2 `improve-scan-card-detection` | #158 | **Shipped** | ❌ not_a_card still ~44%; slight Aug 15 improvement |
|
||
| 3 `scan-visual-catalog-search` | #160, #162 | **Shipped** | ⏳ Awaiting backfill + L0 traffic |
|
||
|
||
**Recommendation:** Finish catalog embedding backfill, scan 50+ cards on
|
||
preview, then re-query `layer = 0` distribution before tuning thresholds.
|
||
|
||
### Embedding backfill status (2026-08-15)
|
||
|
||
| Metric | Value |
|
||
| --- | --- |
|
||
| Cards with `image_url` | 66,211 |
|
||
| Rows with `embedding` | 9,865 (**14.9%**) |
|
||
| pgvector on Neon | ✅ v0.8.0 |
|
||
| Operator command | `npm run backfill-embeddings` |
|
||
|
||
## Worktree
|
||
|
||
| Checkout | Branch | Purpose |
|
||
| --- | --- | --- |
|
||
| `tcg-vault-worktrees/scanner-identify-upgrade` | `convoy/scanner-identify-upgrade` | These convoy docs |
|
||
| (later) Agents Window worktree per brief | `brief/scanner-identify-upgrade/<n>-<title>` | Implementer slices |
|
||
|
||
`scripts/wt.sh` is a deprecation stub. Create implementer worktrees from
|
||
the Agents Window after Architect writes `slice_dependencies:`.
|
||
|
||
## Multitask dispatch
|
||
|
||
No implementer fan-out from this umbrella. After #1 Architect marks
|
||
parallel-safe briefs (`depends_on: []` + disjoint `files:`):
|
||
|
||
```
|
||
/multitask role-implementer briefs <ids>
|
||
```
|
||
|
||
After each PR draft:
|
||
|
||
```
|
||
/multitask role-reviewer + role-security-auditor
|
||
```
|
||
|
||
Add design-system + a11y auditors only if the PR touches
|
||
`components/` or `pages/scanner.js` (not expected in #1).
|
||
|
||
Group ids: `audit-tighten-scan-identify-hot-path-<pr>`,
|
||
`audit-improve-scan-card-detection-<pr>`,
|
||
`audit-scan-visual-catalog-search-<pr>`.
|