deckhearth/.convoys/improve-scan-card-detection.md

name	classification	success_metric	skip	status	created	depends_on	umbrella	model_policy
improve-scan-card-detection	feature	After ship, L2 result_kind=not_a_card ≤15% of L2 (was 44.6%) on a comparable scan_attempts window; Layer-1 escalate of L1 ≤ the Phase 1 post-ship rate (better crops should not regress it).	ia ui-design flag	shipped	2026-08-14	tighten-scan-identify-hot-path	scanner-identify-upgrade	default_session roles escalate_to escalate_to_premium never_premium auto role-conductor role-architect role-ia-architect role-ux-reviewer role-ui-designer role-implementer role-reviewer role-security-auditor role-design-system-auditor role-a11y-auditor role-doc-writer composer-2.5-fast composer-2.5 composer-2.5-fast composer-2.5-fast composer-2.5-fast composer-2.5-fast cursor-grok-4.5-high gpt-5.6-terra-medium cursor-grok-4.5-high cursor-grok-4.5-high auto claude-sonnet-5-thinking-medium claude-4.6-opus-high-thinking role-reviewer role-security-auditor role-design-system-auditor role-a11y-auditor role-ui-designer role-doc-writer

Convoy: improve-scan-card-detection

Phase 2 of scanner-identify-upgrade. Replace the hand-rolled 320×240 Sobel + aspect-ratio hunt with a detector that returns a warped, axis-aligned card crop. Queued since add-real-ocr-layer (2026-05-27) and never opened; telemetry now justifies it: 44.6% of Gemini calls are not_a_card.

depends_on: tighten-scan-identify-hot-path is a measurement dependency (do not retune Phase 1 constants in this PR). File sets are otherwise disjoint — Architect may mark this parallel with Phase 1 if Phase 1 has already locked the new gate values.

Why

detectCardShapesFromFrame is not OpenCV (despite the hook comment). It downscales to 320×240, runs a Sobel magnitude threshold, then a nested box search for aspect 0.63–0.77. There is no four-corner homography. Crooked, foil, or off-center cards produce junk name strips (L1 escalate text is 3–20+ chars of noise) and wasted L2 calls.

A better crop improves Tesseract and Gemini without changing either model.

Scope

In scope

• Replace or wrap detectCardShapesFromFrame so tracked bounds are card-shaped and a perspective-corrected JPEG can be produced for identify.
• Architect picks one in-browser approach (do not land both):
  1. OpenCV.js contours + approxPolyDP + warpPerspective — no training, larger WASM.
  2. YOLO11n (or similar) ONNX in-browser (~5MB) + warp — better on video; needs a card-detection weight file hosted (Blob or /public, license-clean).
• Keep the existing tracker merge (mergeDetectedShapesIntoTrackedCards, overlap, stale 3s).
• Overlay brackets in components/scanner/ScannerCamera.js keep consuming {x,y,width,height} (or four corners mapped to a rect). Do not redesign the overlay.
• Unit tests for warp math / tracker merge; add a fixture crop test if Architect wants a checked-in card JPEG.

Out of scope

• Identify / Gemini / Tesseract / catalog match (Phase 1 + 3).
• Python/CUDA microservice, PaddleOCR server, Roboflow-hosted detect.
• Scanner chrome / cart (scanner-mobile-checkout).
• Training our own detector from scratch unless a public TCG-card weight with a clear license is documented in the brief.

Roles invoked

1. role-ux-reviewer — time-to-bracket, false-positive boxes, dual-card frames.
2. role-architect — OpenCV.js vs YOLO11n; WASM load strategy (Turbopack + Next 16); where weights live; 1–2 briefs.
3. role-implementer.
4. Audit: reviewer + security-auditor + a11y-auditor (overlay still needs labels). Design-system only if overlay styling changes.

Todos

• Architect: pick detector; document WASM / weight budget
• Brief 1 — detect + warp library + tests
• Brief 2 — wire use-camera-scanner.js + crop used by identifyTrackedCardCapture
• Confirm scanner-mobile-checkout overlay still maps bounds
• Re-measure L2 not_a_card share (2026-08-15 — 43.9% all-time; 38.9% on Aug 15 n=26)

Post-ship (PR #158, 2026-08-14)

Shipped: perspective warp + scanner-card-warp.js; warped JPEG feeds OCR and vision paths.

Telemetry: L2 not_a_card 43.9% all-time vs 44.6% baseline — flat. Aug 15 burst (n=26) showed 38.9% (directional improvement only). L1 OCR quality regressed (shorter garbage strips).

Verdict: Better crops shipped; Gemini still sees bad frames often. Combined with Phase 3 backfill for printing-accurate matches.

Likely file ownership

Area	Files
Detect	lib/scanner-card-detection.js, test/lib/scanner-card-detection.test.js
Warp helper	new lib/scanner-card-warp.js (if Architect splits)
Camera loop	lib/use-camera-scanner.js
Crop consumer	lib/scanner-card-identify.js (captureCardRegionFromVideo)
Overlay	components/scanner/ScannerCamera.js only if bounds shape changes

Do not edit lib/ocr-worker.js or lib/scan-vision.js here.

Multitask dispatch

Serial unless Architect splits detect-lib vs camera-wire with disjoint files.

Audit group id: audit-improve-scan-card-detection-<pr>.

CI impact

Workflow / job	Behavior
visual-diff.yml	May fire if ScannerCamera.js changes
preview-smoke.yml	Fires
ci.yml test	New/updated detection tests

Operator action

If YOLO weights are used: confirm license + host on Vercel Blob or public/ (cache-Control immutable). No new secrets.

Conductor notes

Do not drop OpenCV.js in only to reimplement the current rectangle hunt. Success is a rectified card image, not a prettier box. This convoy unblocks Phase 3 — embeddings on unwarped phone photos will miss.

UX

No new routes or screens. Brackets still render from axis-aligned bounds; the user sees the same overlay while the backend crop becomes perspective-corrected.

Timing

• Bracket appearance unchanged (200ms detect interval, 800ms verify gate from Phase 1).
• Warp adds ~20–40ms on capture only — not on the detect loop.

Dual-card frames

Tracker merge behavior unchanged: overlapping boxes collapse to one tracked card. Warp runs per tracked card at verify time.

Failure modes

When corner refinement fails validation, fall back to the axis-aligned margin crop (same as pre-Phase-2). No new error toast.

Architecture

Decision D1 — Pure-JS contour corners + homography warp (not OpenCV.js WASM)

OpenCV.js adds ~8MB WASM and a Turbopack dynamic-import footgun. Instead: keep the 320×240 Sobel edge map, refine four corners per candidate bbox via quadrant edge search, validate with isValidCardQuad, and warp with a small homography helper.

Rejected: YOLO11n ONNX (weight hosting + license review), OpenCV.js (full WASM budget).

Decision D2 — Corners ride on tracked cards

mergeDetectedShapesIntoTrackedCards stores corners alongside bounds. Overlay continues to use bounds only.

Decision D3 — Warp at capture time only

captureCardRegionFromVideo calls warpCardCaptureFromVideo when four video-space corners exist; otherwise axis-aligned crop.

slice_dependencies

Brief	depends_on	files
1 detect + warp	[]	lib/scanner-card-warp.js, lib/scanner-card-detection.js, tests
2 wire capture	[1]	lib/scanner-card-identify.js, lib/use-camera-scanner.js, ScannerCamera.js

Serial implement: Brief 1 → Brief 2.

Audit group id: audit-improve-scan-card-detection-<pr>.