c6a0225e54
Record Aug 15 measurement showing Phases 1–2 missed success targets, Phase 3 code is live with 15% embedding backfill and zero L0 traffic yet. Co-authored-by: Cursor <cursoragent@cursor.com>
218 lines
7.5 KiB
Markdown
218 lines
7.5 KiB
Markdown
---
|
||
name: improve-scan-card-detection
|
||
classification: feature
|
||
success_metric: |
|
||
After ship, L2 result_kind=not_a_card ≤15% of L2 (was 44.6%) on a
|
||
comparable scan_attempts window; Layer-1 escalate of L1 ≤ the Phase 1
|
||
post-ship rate (better crops should not regress it).
|
||
skip:
|
||
- ia
|
||
- ui-design
|
||
- flag
|
||
status: shipped
|
||
created: 2026-08-14
|
||
depends_on:
|
||
- tighten-scan-identify-hot-path
|
||
umbrella: scanner-identify-upgrade
|
||
model_policy:
|
||
default_session: auto
|
||
roles:
|
||
role-conductor: composer-2.5-fast
|
||
role-architect: composer-2.5
|
||
role-ia-architect: composer-2.5-fast
|
||
role-ux-reviewer: composer-2.5-fast
|
||
role-ui-designer: composer-2.5-fast
|
||
role-implementer: composer-2.5-fast
|
||
role-reviewer: cursor-grok-4.5-high
|
||
role-security-auditor: gpt-5.6-terra-medium
|
||
role-design-system-auditor: cursor-grok-4.5-high
|
||
role-a11y-auditor: cursor-grok-4.5-high
|
||
role-doc-writer: auto
|
||
escalate_to: claude-sonnet-5-thinking-medium
|
||
escalate_to_premium: claude-4.6-opus-high-thinking
|
||
never_premium:
|
||
- role-reviewer
|
||
- role-security-auditor
|
||
- role-design-system-auditor
|
||
- role-a11y-auditor
|
||
- role-ui-designer
|
||
- role-doc-writer
|
||
---
|
||
|
||
# Convoy: improve-scan-card-detection
|
||
|
||
Phase 2 of `scanner-identify-upgrade`. Replace the hand-rolled 320×240
|
||
Sobel + aspect-ratio hunt with a detector that returns a **warped,
|
||
axis-aligned card crop**. Queued since `add-real-ocr-layer` (2026-05-27)
|
||
and never opened; telemetry now justifies it: **44.6% of Gemini calls
|
||
are `not_a_card`**.
|
||
|
||
`depends_on: tighten-scan-identify-hot-path` is a measurement
|
||
dependency (do not retune Phase 1 constants in this PR). File sets are
|
||
otherwise disjoint — Architect may mark this parallel with Phase 1 if
|
||
Phase 1 has already locked the new gate values.
|
||
|
||
## Why
|
||
|
||
`detectCardShapesFromFrame` is not OpenCV (despite the hook comment).
|
||
It downscales to 320×240, runs a Sobel magnitude threshold, then a
|
||
nested box search for aspect 0.63–0.77. There is no four-corner
|
||
homography. Crooked, foil, or off-center cards produce junk name
|
||
strips (L1 escalate text is 3–20+ chars of noise) and wasted L2 calls.
|
||
|
||
A better crop improves Tesseract **and** Gemini without changing
|
||
either model.
|
||
|
||
## Scope
|
||
|
||
### In scope
|
||
|
||
- Replace or wrap `detectCardShapesFromFrame` so tracked bounds are
|
||
card-shaped **and** a perspective-corrected JPEG can be produced for
|
||
identify.
|
||
- Architect picks **one** in-browser approach (do not land both):
|
||
1. **OpenCV.js contours + approxPolyDP + warpPerspective** — no
|
||
training, larger WASM.
|
||
2. **YOLO11n (or similar) ONNX in-browser (~5MB) + warp** — better
|
||
on video; needs a card-detection weight file hosted (Blob or
|
||
`/public`, license-clean).
|
||
- Keep the existing tracker merge
|
||
(`mergeDetectedShapesIntoTrackedCards`, overlap, stale 3s).
|
||
- Overlay brackets in `components/scanner/ScannerCamera.js` keep
|
||
consuming `{x,y,width,height}` (or four corners mapped to a rect).
|
||
Do not redesign the overlay.
|
||
- Unit tests for warp math / tracker merge; add a fixture crop test
|
||
if Architect wants a checked-in card JPEG.
|
||
|
||
### Out of scope
|
||
|
||
- Identify / Gemini / Tesseract / catalog match (Phase 1 + 3).
|
||
- Python/CUDA microservice, PaddleOCR server, Roboflow-hosted detect.
|
||
- Scanner chrome / cart (`scanner-mobile-checkout`).
|
||
- Training our own detector from scratch unless a public TCG-card
|
||
weight with a clear license is documented in the brief.
|
||
|
||
## Roles invoked
|
||
|
||
1. `role-ux-reviewer` — time-to-bracket, false-positive boxes, dual-card
|
||
frames.
|
||
2. `role-architect` — OpenCV.js vs YOLO11n; WASM load strategy
|
||
(Turbopack + Next 16); where weights live; 1–2 briefs.
|
||
3. `role-implementer`.
|
||
4. Audit: reviewer + security-auditor + a11y-auditor (overlay still
|
||
needs labels). Design-system only if overlay styling changes.
|
||
|
||
## Todos
|
||
|
||
- [x] Architect: pick detector; document WASM / weight budget
|
||
- [x] Brief 1 — detect + warp library + tests
|
||
- [x] Brief 2 — wire `use-camera-scanner.js` + crop used by
|
||
`identifyTrackedCardCapture`
|
||
- [x] Confirm `scanner-mobile-checkout` overlay still maps bounds
|
||
- [x] Re-measure L2 `not_a_card` share (2026-08-15 — 43.9% all-time; 38.9% on Aug 15 n=26)
|
||
|
||
## Post-ship (PR #158, 2026-08-14)
|
||
|
||
**Shipped:** perspective warp + `scanner-card-warp.js`; warped JPEG feeds OCR
|
||
and vision paths.
|
||
|
||
**Telemetry:** L2 `not_a_card` 43.9% all-time vs 44.6% baseline — flat.
|
||
Aug 15 burst (n=26) showed 38.9% (directional improvement only). L1 OCR
|
||
quality regressed (shorter garbage strips).
|
||
|
||
**Verdict:** Better crops shipped; Gemini still sees bad frames often.
|
||
Combined with Phase 3 backfill for printing-accurate matches.
|
||
|
||
## Likely file ownership
|
||
|
||
| Area | Files |
|
||
| --- | --- |
|
||
| Detect | `lib/scanner-card-detection.js`, `test/lib/scanner-card-detection.test.js` |
|
||
| Warp helper | new `lib/scanner-card-warp.js` (if Architect splits) |
|
||
| Camera loop | `lib/use-camera-scanner.js` |
|
||
| Crop consumer | `lib/scanner-card-identify.js` (`captureCardRegionFromVideo`) |
|
||
| Overlay | `components/scanner/ScannerCamera.js` only if bounds shape changes |
|
||
|
||
Do not edit `lib/ocr-worker.js` or `lib/scan-vision.js` here.
|
||
|
||
## Multitask dispatch
|
||
|
||
Serial unless Architect splits detect-lib vs camera-wire with disjoint
|
||
files.
|
||
|
||
Audit group id: `audit-improve-scan-card-detection-<pr>`.
|
||
|
||
## CI impact
|
||
|
||
| Workflow / job | Behavior |
|
||
| --- | --- |
|
||
| `visual-diff.yml` | **May fire** if `ScannerCamera.js` changes |
|
||
| `preview-smoke.yml` | Fires |
|
||
| `ci.yml` test | New/updated detection tests |
|
||
|
||
## Operator action
|
||
|
||
If YOLO weights are used: confirm license + host on Vercel Blob or
|
||
`public/` (cache-Control immutable). No new secrets.
|
||
|
||
## Conductor notes
|
||
|
||
Do not drop OpenCV.js in only to reimplement the current rectangle
|
||
hunt. Success is a **rectified card image**, not a prettier box.
|
||
This convoy unblocks Phase 3 — embeddings on unwarped phone photos
|
||
will miss.
|
||
|
||
## UX
|
||
|
||
No new routes or screens. Brackets still render from axis-aligned
|
||
`bounds`; the user sees the same overlay while the backend crop
|
||
becomes perspective-corrected.
|
||
|
||
### Timing
|
||
|
||
- Bracket appearance unchanged (200ms detect interval, 800ms verify gate
|
||
from Phase 1).
|
||
- Warp adds ~20–40ms on capture only — not on the detect loop.
|
||
|
||
### Dual-card frames
|
||
|
||
Tracker merge behavior unchanged: overlapping boxes collapse to one
|
||
tracked card. Warp runs per tracked card at verify time.
|
||
|
||
### Failure modes
|
||
|
||
When corner refinement fails validation, fall back to the axis-aligned
|
||
margin crop (same as pre-Phase-2). No new error toast.
|
||
|
||
## Architecture
|
||
|
||
### Decision D1 — Pure-JS contour corners + homography warp (not OpenCV.js WASM)
|
||
|
||
OpenCV.js adds ~8MB WASM and a Turbopack dynamic-import footgun.
|
||
Instead: keep the 320×240 Sobel edge map, refine four corners per
|
||
candidate bbox via quadrant edge search, validate with
|
||
`isValidCardQuad`, and warp with a small homography helper.
|
||
|
||
Rejected: YOLO11n ONNX (weight hosting + license review), OpenCV.js
|
||
(full WASM budget).
|
||
|
||
### Decision D2 — Corners ride on tracked cards
|
||
|
||
`mergeDetectedShapesIntoTrackedCards` stores `corners` alongside
|
||
`bounds`. Overlay continues to use `bounds` only.
|
||
|
||
### Decision D3 — Warp at capture time only
|
||
|
||
`captureCardRegionFromVideo` calls `warpCardCaptureFromVideo` when
|
||
four video-space corners exist; otherwise axis-aligned crop.
|
||
|
||
### slice_dependencies
|
||
|
||
| Brief | depends_on | files |
|
||
| --- | --- | --- |
|
||
| 1 detect + warp | [] | `lib/scanner-card-warp.js`, `lib/scanner-card-detection.js`, tests |
|
||
| 2 wire capture | [1] | `lib/scanner-card-identify.js`, `lib/use-camera-scanner.js`, `ScannerCamera.js` |
|
||
|
||
Serial implement: Brief 1 → Brief 2.
|
||
|
||
Audit group id: `audit-improve-scan-card-detection-<pr>`.
|