--- name: improve-scan-card-detection classification: feature success_metric: | After ship, L2 result_kind=not_a_card ≤15% of L2 (was 44.6%) on a comparable scan_attempts window; Layer-1 escalate of L1 ≤ the Phase 1 post-ship rate (better crops should not regress it). skip: - ia - ui-design - flag status: shipped created: 2026-08-14 depends_on: - tighten-scan-identify-hot-path umbrella: scanner-identify-upgrade model_policy: default_session: auto roles: role-conductor: composer-2.5-fast role-architect: composer-2.5 role-ia-architect: composer-2.5-fast role-ux-reviewer: composer-2.5-fast role-ui-designer: composer-2.5-fast role-implementer: composer-2.5-fast role-reviewer: cursor-grok-4.5-high role-security-auditor: gpt-5.6-terra-medium role-design-system-auditor: cursor-grok-4.5-high role-a11y-auditor: cursor-grok-4.5-high role-doc-writer: auto escalate_to: claude-sonnet-5-thinking-medium escalate_to_premium: claude-4.6-opus-high-thinking never_premium: - role-reviewer - role-security-auditor - role-design-system-auditor - role-a11y-auditor - role-ui-designer - role-doc-writer --- # Convoy: improve-scan-card-detection Phase 2 of `scanner-identify-upgrade`. Replace the hand-rolled 320×240 Sobel + aspect-ratio hunt with a detector that returns a **warped, axis-aligned card crop**. Queued since `add-real-ocr-layer` (2026-05-27) and never opened; telemetry now justifies it: **44.6% of Gemini calls are `not_a_card`**. `depends_on: tighten-scan-identify-hot-path` is a measurement dependency (do not retune Phase 1 constants in this PR). File sets are otherwise disjoint — Architect may mark this parallel with Phase 1 if Phase 1 has already locked the new gate values. ## Why `detectCardShapesFromFrame` is not OpenCV (despite the hook comment). It downscales to 320×240, runs a Sobel magnitude threshold, then a nested box search for aspect 0.63–0.77. There is no four-corner homography. Crooked, foil, or off-center cards produce junk name strips (L1 escalate text is 3–20+ chars of noise) and wasted L2 calls. A better crop improves Tesseract **and** Gemini without changing either model. ## Scope ### In scope - Replace or wrap `detectCardShapesFromFrame` so tracked bounds are card-shaped **and** a perspective-corrected JPEG can be produced for identify. - Architect picks **one** in-browser approach (do not land both): 1. **OpenCV.js contours + approxPolyDP + warpPerspective** — no training, larger WASM. 2. **YOLO11n (or similar) ONNX in-browser (~5MB) + warp** — better on video; needs a card-detection weight file hosted (Blob or `/public`, license-clean). - Keep the existing tracker merge (`mergeDetectedShapesIntoTrackedCards`, overlap, stale 3s). - Overlay brackets in `components/scanner/ScannerCamera.js` keep consuming `{x,y,width,height}` (or four corners mapped to a rect). Do not redesign the overlay. - Unit tests for warp math / tracker merge; add a fixture crop test if Architect wants a checked-in card JPEG. ### Out of scope - Identify / Gemini / Tesseract / catalog match (Phase 1 + 3). - Python/CUDA microservice, PaddleOCR server, Roboflow-hosted detect. - Scanner chrome / cart (`scanner-mobile-checkout`). - Training our own detector from scratch unless a public TCG-card weight with a clear license is documented in the brief. ## Roles invoked 1. `role-ux-reviewer` — time-to-bracket, false-positive boxes, dual-card frames. 2. `role-architect` — OpenCV.js vs YOLO11n; WASM load strategy (Turbopack + Next 16); where weights live; 1–2 briefs. 3. `role-implementer`. 4. Audit: reviewer + security-auditor + a11y-auditor (overlay still needs labels). Design-system only if overlay styling changes. ## Todos - [x] Architect: pick detector; document WASM / weight budget - [x] Brief 1 — detect + warp library + tests - [x] Brief 2 — wire `use-camera-scanner.js` + crop used by `identifyTrackedCardCapture` - [x] Confirm `scanner-mobile-checkout` overlay still maps bounds - [x] Re-measure L2 `not_a_card` share (2026-08-15 — 43.9% all-time; 38.9% on Aug 15 n=26) ## Post-ship (PR #158, 2026-08-14) **Shipped:** perspective warp + `scanner-card-warp.js`; warped JPEG feeds OCR and vision paths. **Telemetry:** L2 `not_a_card` 43.9% all-time vs 44.6% baseline — flat. Aug 15 burst (n=26) showed 38.9% (directional improvement only). L1 OCR quality regressed (shorter garbage strips). **Verdict:** Better crops shipped; Gemini still sees bad frames often. Combined with Phase 3 backfill for printing-accurate matches. ## Likely file ownership | Area | Files | | --- | --- | | Detect | `lib/scanner-card-detection.js`, `test/lib/scanner-card-detection.test.js` | | Warp helper | new `lib/scanner-card-warp.js` (if Architect splits) | | Camera loop | `lib/use-camera-scanner.js` | | Crop consumer | `lib/scanner-card-identify.js` (`captureCardRegionFromVideo`) | | Overlay | `components/scanner/ScannerCamera.js` only if bounds shape changes | Do not edit `lib/ocr-worker.js` or `lib/scan-vision.js` here. ## Multitask dispatch Serial unless Architect splits detect-lib vs camera-wire with disjoint files. Audit group id: `audit-improve-scan-card-detection-`. ## CI impact | Workflow / job | Behavior | | --- | --- | | `visual-diff.yml` | **May fire** if `ScannerCamera.js` changes | | `preview-smoke.yml` | Fires | | `ci.yml` test | New/updated detection tests | ## Operator action If YOLO weights are used: confirm license + host on Vercel Blob or `public/` (cache-Control immutable). No new secrets. ## Conductor notes Do not drop OpenCV.js in only to reimplement the current rectangle hunt. Success is a **rectified card image**, not a prettier box. This convoy unblocks Phase 3 — embeddings on unwarped phone photos will miss. ## UX No new routes or screens. Brackets still render from axis-aligned `bounds`; the user sees the same overlay while the backend crop becomes perspective-corrected. ### Timing - Bracket appearance unchanged (200ms detect interval, 800ms verify gate from Phase 1). - Warp adds ~20–40ms on capture only — not on the detect loop. ### Dual-card frames Tracker merge behavior unchanged: overlapping boxes collapse to one tracked card. Warp runs per tracked card at verify time. ### Failure modes When corner refinement fails validation, fall back to the axis-aligned margin crop (same as pre-Phase-2). No new error toast. ## Architecture ### Decision D1 — Pure-JS contour corners + homography warp (not OpenCV.js WASM) OpenCV.js adds ~8MB WASM and a Turbopack dynamic-import footgun. Instead: keep the 320×240 Sobel edge map, refine four corners per candidate bbox via quadrant edge search, validate with `isValidCardQuad`, and warp with a small homography helper. Rejected: YOLO11n ONNX (weight hosting + license review), OpenCV.js (full WASM budget). ### Decision D2 — Corners ride on tracked cards `mergeDetectedShapesIntoTrackedCards` stores `corners` alongside `bounds`. Overlay continues to use `bounds` only. ### Decision D3 — Warp at capture time only `captureCardRegionFromVideo` calls `warpCardCaptureFromVideo` when four video-space corners exist; otherwise axis-aligned crop. ### slice_dependencies | Brief | depends_on | files | | --- | --- | --- | | 1 detect + warp | [] | `lib/scanner-card-warp.js`, `lib/scanner-card-detection.js`, tests | | 2 wire capture | [1] | `lib/scanner-card-identify.js`, `lib/use-camera-scanner.js`, `ScannerCamera.js` | Serial implement: Brief 1 → Brief 2. Audit group id: `audit-improve-scan-card-detection-`.