152 lines
5.2 KiB
Markdown
152 lines
5.2 KiB
Markdown
|
---
|
|||
|
|
name: improve-scan-card-detection
|
|||
|
|
classification: feature
|
|||
|
|
success_metric: |
|
|||
|
|
After ship, L2 result_kind=not_a_card ≤15% of L2 (was 44.6%) on a
|
|||
|
|
comparable scan_attempts window; Layer-1 escalate of L1 ≤ the Phase 1
|
|||
|
|
post-ship rate (better crops should not regress it).
|
|||
|
|
skip:
|
|||
|
|
- ia
|
|||
|
|
- ui-design
|
|||
|
|
- flag
|
|||
|
|
status: open
|
|||
|
|
created: 2026-08-14
|
|||
|
|
depends_on:
|
|||
|
|
- tighten-scan-identify-hot-path
|
|||
|
|
umbrella: scanner-identify-upgrade
|
|||
|
|
model_policy:
|
|||
|
|
default_session: auto
|
|||
|
|
roles:
|
|||
|
|
role-conductor: composer-2.5-fast
|
|||
|
|
role-architect: composer-2.5
|
|||
|
|
role-ia-architect: composer-2.5-fast
|
|||
|
|
role-ux-reviewer: composer-2.5-fast
|
|||
|
|
role-ui-designer: composer-2.5-fast
|
|||
|
|
role-implementer: composer-2.5-fast
|
|||
|
|
role-reviewer: cursor-grok-4.5-high
|
|||
|
|
role-security-auditor: gpt-5.6-terra-medium
|
|||
|
|
role-design-system-auditor: cursor-grok-4.5-high
|
|||
|
|
role-a11y-auditor: cursor-grok-4.5-high
|
|||
|
|
role-doc-writer: auto
|
|||
|
|
escalate_to: claude-sonnet-5-thinking-medium
|
|||
|
|
escalate_to_premium: claude-4.6-opus-high-thinking
|
|||
|
|
never_premium:
|
|||
|
|
- role-reviewer
|
|||
|
|
- role-security-auditor
|
|||
|
|
- role-design-system-auditor
|
|||
|
|
- role-a11y-auditor
|
|||
|
|
- role-ui-designer
|
|||
|
|
- role-doc-writer
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Convoy: improve-scan-card-detection
|
|||
|
|
|
|||
|
|
Phase 2 of `scanner-identify-upgrade`. Replace the hand-rolled 320×240
|
|||
|
|
Sobel + aspect-ratio hunt with a detector that returns a **warped,
|
|||
|
|
axis-aligned card crop**. Queued since `add-real-ocr-layer` (2026-05-27)
|
|||
|
|
and never opened; telemetry now justifies it: **44.6% of Gemini calls
|
|||
|
|
are `not_a_card`**.
|
|||
|
|
|
|||
|
|
`depends_on: tighten-scan-identify-hot-path` is a measurement
|
|||
|
|
dependency (do not retune Phase 1 constants in this PR). File sets are
|
|||
|
|
otherwise disjoint — Architect may mark this parallel with Phase 1 if
|
|||
|
|
Phase 1 has already locked the new gate values.
|
|||
|
|
|
|||
|
|
## Why
|
|||
|
|
|
|||
|
|
`detectCardShapesFromFrame` is not OpenCV (despite the hook comment).
|
|||
|
|
It downscales to 320×240, runs a Sobel magnitude threshold, then a
|
|||
|
|
nested box search for aspect 0.63–0.77. There is no four-corner
|
|||
|
|
homography. Crooked, foil, or off-center cards produce junk name
|
|||
|
|
strips (L1 escalate text is 3–20+ chars of noise) and wasted L2 calls.
|
|||
|
|
|
|||
|
|
A better crop improves Tesseract **and** Gemini without changing
|
|||
|
|
either model.
|
|||
|
|
|
|||
|
|
## Scope
|
|||
|
|
|
|||
|
|
### In scope
|
|||
|
|
|
|||
|
|
- Replace or wrap `detectCardShapesFromFrame` so tracked bounds are
|
|||
|
|
card-shaped **and** a perspective-corrected JPEG can be produced for
|
|||
|
|
identify.
|
|||
|
|
- Architect picks **one** in-browser approach (do not land both):
|
|||
|
|
1. **OpenCV.js contours + approxPolyDP + warpPerspective** — no
|
|||
|
|
training, larger WASM.
|
|||
|
|
2. **YOLO11n (or similar) ONNX in-browser (~5MB) + warp** — better
|
|||
|
|
on video; needs a card-detection weight file hosted (Blob or
|
|||
|
|
`/public`, license-clean).
|
|||
|
|
- Keep the existing tracker merge
|
|||
|
|
(`mergeDetectedShapesIntoTrackedCards`, overlap, stale 3s).
|
|||
|
|
- Overlay brackets in `components/scanner/ScannerCamera.js` keep
|
|||
|
|
consuming `{x,y,width,height}` (or four corners mapped to a rect).
|
|||
|
|
Do not redesign the overlay.
|
|||
|
|
- Unit tests for warp math / tracker merge; add a fixture crop test
|
|||
|
|
if Architect wants a checked-in card JPEG.
|
|||
|
|
|
|||
|
|
### Out of scope
|
|||
|
|
|
|||
|
|
- Identify / Gemini / Tesseract / catalog match (Phase 1 + 3).
|
|||
|
|
- Python/CUDA microservice, PaddleOCR server, Roboflow-hosted detect.
|
|||
|
|
- Scanner chrome / cart (`scanner-mobile-checkout`).
|
|||
|
|
- Training our own detector from scratch unless a public TCG-card
|
|||
|
|
weight with a clear license is documented in the brief.
|
|||
|
|
|
|||
|
|
## Roles invoked
|
|||
|
|
|
|||
|
|
1. `role-ux-reviewer` — time-to-bracket, false-positive boxes, dual-card
|
|||
|
|
frames.
|
|||
|
|
2. `role-architect` — OpenCV.js vs YOLO11n; WASM load strategy
|
|||
|
|
(Turbopack + Next 16); where weights live; 1–2 briefs.
|
|||
|
|
3. `role-implementer`.
|
|||
|
|
4. Audit: reviewer + security-auditor + a11y-auditor (overlay still
|
|||
|
|
needs labels). Design-system only if overlay styling changes.
|
|||
|
|
|
|||
|
|
## Todos
|
|||
|
|
|
|||
|
|
- [ ] Architect: pick detector; document WASM / weight budget
|
|||
|
|
- [ ] Brief 1 — detect + warp library + tests
|
|||
|
|
- [ ] Brief 2 — wire `use-camera-scanner.js` + crop used by
|
|||
|
|
`identifyTrackedCardCapture`
|
|||
|
|
- [ ] Confirm `scanner-mobile-checkout` overlay still maps bounds
|
|||
|
|
- [ ] Re-measure L2 `not_a_card` share
|
|||
|
|
|
|||
|
|
## Likely file ownership
|
|||
|
|
|
|||
|
|
| Area | Files |
|
|||
|
|
| --- | --- |
|
|||
|
|
| Detect | `lib/scanner-card-detection.js`, `test/lib/scanner-card-detection.test.js` |
|
|||
|
|
| Warp helper | new `lib/scanner-card-warp.js` (if Architect splits) |
|
|||
|
|
| Camera loop | `lib/use-camera-scanner.js` |
|
|||
|
|
| Crop consumer | `lib/scanner-card-identify.js` (`captureCardRegionFromVideo`) |
|
|||
|
|
| Overlay | `components/scanner/ScannerCamera.js` only if bounds shape changes |
|
|||
|
|
|
|||
|
|
Do not edit `lib/ocr-worker.js` or `lib/scan-vision.js` here.
|
|||
|
|
|
|||
|
|
## Multitask dispatch
|
|||
|
|
|
|||
|
|
Serial unless Architect splits detect-lib vs camera-wire with disjoint
|
|||
|
|
files.
|
|||
|
|
|
|||
|
|
Audit group id: `audit-improve-scan-card-detection-<pr>`.
|
|||
|
|
|
|||
|
|
## CI impact
|
|||
|
|
|
|||
|
|
| Workflow / job | Behavior |
|
|||
|
|
| --- | --- |
|
|||
|
|
| `visual-diff.yml` | **May fire** if `ScannerCamera.js` changes |
|
|||
|
|
| `preview-smoke.yml` | Fires |
|
|||
|
|
| `ci.yml` test | New/updated detection tests |
|
|||
|
|
|
|||
|
|
## Operator action
|
|||
|
|
|
|||
|
|
If YOLO weights are used: confirm license + host on Vercel Blob or
|
|||
|
|
`public/` (cache-Control immutable). No new secrets.
|
|||
|
|
|
|||
|
|
## Conductor notes
|
|||
|
|
|
|||
|
|
Do not drop OpenCV.js in only to reimplement the current rectangle
|
|||
|
|
hunt. Success is a **rectified card image**, not a prettier box.
|
|||
|
|
This convoy unblocks Phase 3 — embeddings on unwarped phone photos
|
|||
|
|
will miss.
|