rstillwell/deckhearth
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions
deckhearth/.convoys/improve-scan-card-detection.md
varutasu 0b4f419f49
Scanner identify upgrade — Phase 1 hot path (#156) 
* docs(convoy): seed scanner identify upgrade epic and sub-convoys

Baseline scan_attempts telemetry and three-phase plan for faster, more
accurate card identification without touching scanner chrome.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(scanner): tighten Layer-1 identify hot path (Phase 1)

Cut verify hold-still gates, OCR collector numbers on Layer 1, request
structured Gemini JSON, and skip automatic L2 refine when L1 opens the
printing picker. Includes convoy UX/architecture briefs and unit tests.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-14 20:21:20 -05:00

5.2 KiB
Raw Blame History

name classification success_metric skip status created depends_on umbrella model_policy
improve-scan-card-detection feature After ship, L2 result_kind=not_a_card ≤15% of L2 (was 44.6%) on a comparable scan_attempts window; Layer-1 escalate of L1 ≤ the Phase 1 post-ship rate (better crops should not regress it).
ia
ui-design
flag
open 2026-08-14
tighten-scan-identify-hot-path
scanner-identify-upgrade
default_session roles escalate_to escalate_to_premium never_premium
auto
role-conductor role-architect role-ia-architect role-ux-reviewer role-ui-designer role-implementer role-reviewer role-security-auditor role-design-system-auditor role-a11y-auditor role-doc-writer
composer-2.5-fast composer-2.5 composer-2.5-fast composer-2.5-fast composer-2.5-fast composer-2.5-fast cursor-grok-4.5-high gpt-5.6-terra-medium cursor-grok-4.5-high cursor-grok-4.5-high auto
claude-sonnet-5-thinking-medium claude-4.6-opus-high-thinking
role-reviewer
role-security-auditor
role-design-system-auditor
role-a11y-auditor
role-ui-designer
role-doc-writer

Convoy: improve-scan-card-detection

Phase 2 of scanner-identify-upgrade. Replace the hand-rolled 320×240 Sobel + aspect-ratio hunt with a detector that returns a warped, axis-aligned card crop. Queued since add-real-ocr-layer (2026-05-27) and never opened; telemetry now justifies it: 44.6% of Gemini calls are not_a_card.

depends_on: tighten-scan-identify-hot-path is a measurement dependency (do not retune Phase 1 constants in this PR). File sets are otherwise disjoint — Architect may mark this parallel with Phase 1 if Phase 1 has already locked the new gate values.

Why

detectCardShapesFromFrame is not OpenCV (despite the hook comment). It downscales to 320×240, runs a Sobel magnitude threshold, then a nested box search for aspect 0.63–0.77. There is no four-corner homography. Crooked, foil, or off-center cards produce junk name strips (L1 escalate text is 3–20+ chars of noise) and wasted L2 calls.

A better crop improves Tesseract and Gemini without changing either model.

Scope

In scope

  • Replace or wrap detectCardShapesFromFrame so tracked bounds are card-shaped and a perspective-corrected JPEG can be produced for identify.
  • Architect picks one in-browser approach (do not land both):
    1. OpenCV.js contours + approxPolyDP + warpPerspective — no training, larger WASM.
    2. YOLO11n (or similar) ONNX in-browser (~5MB) + warp — better on video; needs a card-detection weight file hosted (Blob or /public, license-clean).
  • Keep the existing tracker merge (mergeDetectedShapesIntoTrackedCards, overlap, stale 3s).
  • Overlay brackets in components/scanner/ScannerCamera.js keep consuming {x,y,width,height} (or four corners mapped to a rect). Do not redesign the overlay.
  • Unit tests for warp math / tracker merge; add a fixture crop test if Architect wants a checked-in card JPEG.

Out of scope

  • Identify / Gemini / Tesseract / catalog match (Phase 1 + 3).
  • Python/CUDA microservice, PaddleOCR server, Roboflow-hosted detect.
  • Scanner chrome / cart (scanner-mobile-checkout).
  • Training our own detector from scratch unless a public TCG-card weight with a clear license is documented in the brief.

Roles invoked

  1. role-ux-reviewer — time-to-bracket, false-positive boxes, dual-card frames.
  2. role-architect — OpenCV.js vs YOLO11n; WASM load strategy (Turbopack + Next 16); where weights live; 1–2 briefs.
  3. role-implementer.
  4. Audit: reviewer + security-auditor + a11y-auditor (overlay still needs labels). Design-system only if overlay styling changes.

Todos

  • Architect: pick detector; document WASM / weight budget
  • Brief 1 — detect + warp library + tests
  • Brief 2 — wire use-camera-scanner.js + crop used by identifyTrackedCardCapture
  • Confirm scanner-mobile-checkout overlay still maps bounds
  • Re-measure L2 not_a_card share

Likely file ownership

Area Files
Detect lib/scanner-card-detection.js, test/lib/scanner-card-detection.test.js
Warp helper new lib/scanner-card-warp.js (if Architect splits)
Camera loop lib/use-camera-scanner.js
Crop consumer lib/scanner-card-identify.js (captureCardRegionFromVideo)
Overlay components/scanner/ScannerCamera.js only if bounds shape changes

Do not edit lib/ocr-worker.js or lib/scan-vision.js here.

Multitask dispatch

Serial unless Architect splits detect-lib vs camera-wire with disjoint files.

Audit group id: audit-improve-scan-card-detection-<pr>.

CI impact

Workflow / job Behavior
visual-diff.yml May fire if ScannerCamera.js changes
preview-smoke.yml Fires
ci.yml test New/updated detection tests

Operator action

If YOLO weights are used: confirm license + host on Vercel Blob or public/ (cache-Control immutable). No new secrets.

Conductor notes

Do not drop OpenCV.js in only to reimplement the current rectangle hunt. Success is a rectified card image, not a prettier box. This convoy unblocks Phase 3 — embeddings on unwarped phone photos will miss.