deckhearth/.convoys/improve-scan-card-detection.md

---
name: improve-scan-card-detection
classification: feature
success_metric: |
  After ship, L2 result_kind=not_a_card ≤15% of L2 (was 44.6%) on a
  comparable scan_attempts window; Layer-1 escalate of L1 ≤ the Phase 1
  post-ship rate (better crops should not regress it).
skip:
  - ia
  - ui-design
  - flag
status: shipped
created: 2026-08-14
depends_on:
  - tighten-scan-identify-hot-path
umbrella: scanner-identify-upgrade
model_policy:
  default_session: auto
  roles:
    role-conductor: composer-2.5-fast
    role-architect: composer-2.5
    role-ia-architect: composer-2.5-fast
    role-ux-reviewer: composer-2.5-fast
    role-ui-designer: composer-2.5-fast
    role-implementer: composer-2.5-fast
    role-reviewer: cursor-grok-4.5-high
    role-security-auditor: gpt-5.6-terra-medium
    role-design-system-auditor: cursor-grok-4.5-high
    role-a11y-auditor: cursor-grok-4.5-high
    role-doc-writer: auto
  escalate_to: claude-sonnet-5-thinking-medium
  escalate_to_premium: claude-4.6-opus-high-thinking
  never_premium:
    - role-reviewer
    - role-security-auditor
    - role-design-system-auditor
    - role-a11y-auditor
    - role-ui-designer
    - role-doc-writer
---

# Convoy: improve-scan-card-detection

Phase 2 of `scanner-identify-upgrade`. Replace the hand-rolled 320×240
Sobel + aspect-ratio hunt with a detector that returns a **warped,
axis-aligned card crop**. Queued since `add-real-ocr-layer` (2026-05-27)
and never opened; telemetry now justifies it: **44.6% of Gemini calls
are `not_a_card`**.

`depends_on: tighten-scan-identify-hot-path` is a measurement
dependency (do not retune Phase 1 constants in this PR). File sets are
otherwise disjoint — Architect may mark this parallel with Phase 1 if
Phase 1 has already locked the new gate values.

## Why

`detectCardShapesFromFrame` is not OpenCV (despite the hook comment).
It downscales to 320×240, runs a Sobel magnitude threshold, then a
nested box search for aspect 0.63–0.77. There is no four-corner
homography. Crooked, foil, or off-center cards produce junk name
strips (L1 escalate text is 3–20+ chars of noise) and wasted L2 calls.

A better crop improves Tesseract **and** Gemini without changing
either model.

## Scope

### In scope

- Replace or wrap `detectCardShapesFromFrame` so tracked bounds are
  card-shaped **and** a perspective-corrected JPEG can be produced for
  identify.
- Architect picks **one** in-browser approach (do not land both):
  1. **OpenCV.js contours + approxPolyDP + warpPerspective** — no
     training, larger WASM.
  2. **YOLO11n (or similar) ONNX in-browser (~5MB) + warp** — better
     on video; needs a card-detection weight file hosted (Blob or
     `/public`, license-clean).
- Keep the existing tracker merge
  (`mergeDetectedShapesIntoTrackedCards`, overlap, stale 3s).
- Overlay brackets in `components/scanner/ScannerCamera.js` keep
  consuming `{x,y,width,height}` (or four corners mapped to a rect).
  Do not redesign the overlay.
- Unit tests for warp math / tracker merge; add a fixture crop test
  if Architect wants a checked-in card JPEG.

### Out of scope

- Identify / Gemini / Tesseract / catalog match (Phase 1 + 3).
- Python/CUDA microservice, PaddleOCR server, Roboflow-hosted detect.
- Scanner chrome / cart (`scanner-mobile-checkout`).
- Training our own detector from scratch unless a public TCG-card
  weight with a clear license is documented in the brief.

## Roles invoked

1. `role-ux-reviewer` — time-to-bracket, false-positive boxes, dual-card
   frames.
2. `role-architect` — OpenCV.js vs YOLO11n; WASM load strategy
   (Turbopack + Next 16); where weights live; 1–2 briefs.
3. `role-implementer`.
4. Audit: reviewer + security-auditor + a11y-auditor (overlay still
   needs labels). Design-system only if overlay styling changes.

## Todos

- [x] Architect: pick detector; document WASM / weight budget
- [x] Brief 1 — detect + warp library + tests
- [x] Brief 2 — wire `use-camera-scanner.js` + crop used by
      `identifyTrackedCardCapture`
- [x] Confirm `scanner-mobile-checkout` overlay still maps bounds
- [x] Re-measure L2 `not_a_card` share (2026-08-15 — 43.9% all-time; 38.9% on Aug 15 n=26)

## Post-ship (PR #158, 2026-08-14)

**Shipped:** perspective warp + `scanner-card-warp.js`; warped JPEG feeds OCR
and vision paths.

**Telemetry:** L2 `not_a_card` 43.9% all-time vs 44.6% baseline — flat.
Aug 15 burst (n=26) showed 38.9% (directional improvement only). L1 OCR
quality regressed (shorter garbage strips).

**Verdict:** Better crops shipped; Gemini still sees bad frames often.
Combined with Phase 3 backfill for printing-accurate matches.

## Likely file ownership

| Area | Files |
| --- | --- |
| Detect | `lib/scanner-card-detection.js`, `test/lib/scanner-card-detection.test.js` |
| Warp helper | new `lib/scanner-card-warp.js` (if Architect splits) |
| Camera loop | `lib/use-camera-scanner.js` |
| Crop consumer | `lib/scanner-card-identify.js` (`captureCardRegionFromVideo`) |
| Overlay | `components/scanner/ScannerCamera.js` only if bounds shape changes |

Do not edit `lib/ocr-worker.js` or `lib/scan-vision.js` here.

## Multitask dispatch

Serial unless Architect splits detect-lib vs camera-wire with disjoint
files.

Audit group id: `audit-improve-scan-card-detection-<pr>`.

## CI impact

| Workflow / job | Behavior |
| --- | --- |
| `visual-diff.yml` | **May fire** if `ScannerCamera.js` changes |
| `preview-smoke.yml` | Fires |
| `ci.yml` test | New/updated detection tests |

## Operator action

If YOLO weights are used: confirm license + host on Vercel Blob or
`public/` (cache-Control immutable). No new secrets.

## Conductor notes

Do not drop OpenCV.js in only to reimplement the current rectangle
hunt. Success is a **rectified card image**, not a prettier box.
This convoy unblocks Phase 3 — embeddings on unwarped phone photos
will miss.

## UX

No new routes or screens. Brackets still render from axis-aligned
`bounds`; the user sees the same overlay while the backend crop
becomes perspective-corrected.

### Timing

- Bracket appearance unchanged (200ms detect interval, 800ms verify gate
  from Phase 1).
- Warp adds ~20–40ms on capture only — not on the detect loop.

### Dual-card frames

Tracker merge behavior unchanged: overlapping boxes collapse to one
tracked card. Warp runs per tracked card at verify time.

### Failure modes

When corner refinement fails validation, fall back to the axis-aligned
margin crop (same as pre-Phase-2). No new error toast.

## Architecture

### Decision D1 — Pure-JS contour corners + homography warp (not OpenCV.js WASM)

OpenCV.js adds ~8MB WASM and a Turbopack dynamic-import footgun.
Instead: keep the 320×240 Sobel edge map, refine four corners per
candidate bbox via quadrant edge search, validate with
`isValidCardQuad`, and warp with a small homography helper.

Rejected: YOLO11n ONNX (weight hosting + license review), OpenCV.js
(full WASM budget).

### Decision D2 — Corners ride on tracked cards

`mergeDetectedShapesIntoTrackedCards` stores `corners` alongside
`bounds`. Overlay continues to use `bounds` only.

### Decision D3 — Warp at capture time only

`captureCardRegionFromVideo` calls `warpCardCaptureFromVideo` when
four video-space corners exist; otherwise axis-aligned crop.

### slice_dependencies

| Brief | depends_on | files |
| --- | --- | --- |
| 1 detect + warp | [] | `lib/scanner-card-warp.js`, `lib/scanner-card-detection.js`, tests |
| 2 wire capture | [1] | `lib/scanner-card-identify.js`, `lib/use-camera-scanner.js`, `ScannerCamera.js` |

Serial implement: Brief 1 → Brief 2.

Audit group id: `audit-improve-scan-card-detection-<pr>`.