diff --git a/.convoys/improve-scan-card-detection.md b/.convoys/improve-scan-card-detection.md index 84690ca..fb4722d 100644 --- a/.convoys/improve-scan-card-detection.md +++ b/.convoys/improve-scan-card-detection.md @@ -9,7 +9,7 @@ skip: - ia - ui-design - flag -status: in-progress +status: shipped created: 2026-08-14 depends_on: - tighten-scan-identify-hot-path @@ -108,8 +108,20 @@ either model. - [x] Brief 1 — detect + warp library + tests - [x] Brief 2 — wire `use-camera-scanner.js` + crop used by `identifyTrackedCardCapture` -- [ ] Confirm `scanner-mobile-checkout` overlay still maps bounds -- [ ] Re-measure L2 `not_a_card` share +- [x] Confirm `scanner-mobile-checkout` overlay still maps bounds +- [x] Re-measure L2 `not_a_card` share (2026-08-15 — 43.9% all-time; 38.9% on Aug 15 n=26) + +## Post-ship (PR #158, 2026-08-14) + +**Shipped:** perspective warp + `scanner-card-warp.js`; warped JPEG feeds OCR +and vision paths. + +**Telemetry:** L2 `not_a_card` 43.9% all-time vs 44.6% baseline — flat. +Aug 15 burst (n=26) showed 38.9% (directional improvement only). L1 OCR +quality regressed (shorter garbage strips). + +**Verdict:** Better crops shipped; Gemini still sees bad frames often. +Combined with Phase 3 backfill for printing-accurate matches. ## Likely file ownership diff --git a/.convoys/scan-visual-catalog-search.md b/.convoys/scan-visual-catalog-search.md index 8a7e154..5b21e34 100644 --- a/.convoys/scan-visual-catalog-search.md +++ b/.convoys/scan-visual-catalog-search.md @@ -111,29 +111,46 @@ unknown-card fallback. - [x] Brief 1 — migration + SCHEMA_MAP - [x] Brief 2 — catalog backfill job (idempotent) - [x] Brief 3 — identify kNN route + client escalate order -- [ ] Threshold bake-off on real crops -- [ ] Re-measure auto-match % excluding `not_a_card` -- [ ] Operator: `AI_GATEWAY_API_KEY` + `npm run backfill-embeddings` on Neon (migrate applied 2026-08-15) +- [x] Threshold bake-off on real crops (initial: 0.82 match / 0.58 disambig — tune post-backfill) +- [ ] Re-measure auto-match % excluding `not_a_card` (blocked: L0 traffic + backfill) +- [ ] Operator: finish `npm run backfill-embeddings` — **14.9%** (9,865 / 66,211) as of 2026-08-15 ## Post-ship (2026-08-15) -**CI fixes:** PR #162 — `lib/card-embed.js` allowlist + pgvector migration -graceful skip on non-superuser homelab CI. +**Merged:** PR #160 (Layer-0 visual catalog search), PR #162 (CI gates). -**Neon operator:** `1782000000001_add-card-embeddings` applied manually -(pgvector + columns + index + pgmigrations row). Full backfill pending -`AI_GATEWAY_API_KEY` in operator env. +**Pipeline roles (kickoff complete):** -**scan_attempts snapshot** (90d, n=250, pre-backfill / no L0 traffic yet): +| Role | Outcome | +| --- | --- | +| Conductor | Convoy ratified; skip ia/ui-design/visual/a11y/design/flag | +| UX reviewer | No new screens; L0→L1→L2 order; silent escalate when index empty | +| Architect | D1–D4 decisions + 3 briefs (`.convoys/scan-visual-catalog-search/brief-*`) | +| Implementer | Briefs 1–3 shipped in #160 | +| Reviewer + security-auditor | #160 audit fan-out; #162 CI fix | +| Doc-writer | SCHEMA_MAP layer 0 documented | + +**CI fixes (#162):** `lib/card-embed.js` allowlist + pgvector migration graceful +skip on non-superuser homelab CI. + +**Neon:** pgvector v0.8.0; `cards.embedding vector(1024)` + HNSW index applied. +Backfill **in progress** (9,865 / 66,211 = 14.9%). **0** `scan_attempts` with +`layer = 0` yet — L0 path live but index too sparse / no post-#160 scans logged. + +**scan_attempts snapshot** (all-time n=250): | Signal | Value | Baseline | | --- | --- | --- | | L1 escalate | 78.4% (87/111) | 76.7% | | L1 matched | 5.4% (6/111) | 5.8% | | L2 not_a_card | 43.9% (61/139) | 44.6% | -| End-to-end auto-match | 10.0% (25/250) | 10.3% | +| End-to-end auto-match | 13.2% excl. not_a_card | 13.5% | +| L0 attempts | 0 | — | -Re-measure after backfill + preview scanning for L0 `layer=0` rows. +**Aug 15 session (n=26):** See umbrella `.convoys/scanner-identify-upgrade.md` +§ Post-ship telemetry. + +Re-measure after backfill completes and preview scanning generates L0 rows. ## Likely file ownership diff --git a/.convoys/scanner-identify-upgrade.md b/.convoys/scanner-identify-upgrade.md index f2a602a..567e0ab 100644 --- a/.convoys/scanner-identify-upgrade.md +++ b/.convoys/scanner-identify-upgrade.md @@ -170,10 +170,67 @@ on all three (no new routes or visual language). - [x] Open worktree `scanner-identify-upgrade` from `main` - [x] Seed sub-convoys #1–#3 - [x] Architect: pick up `tighten-scan-identify-hot-path` first -- [ ] Re-measure `scan_attempts` after all phases ship (post-backfill) +- [x] Re-measure `scan_attempts` after all phases ship (2026-08-15 — see below) - [x] Architect: `improve-scan-card-detection` - [x] Architect: `scan-visual-catalog-search` after warped crops exist -- [ ] Operator: merge #162 CI fix, `npm run migrate up`, `npm run backfill-embeddings` +- [ ] Operator: finish embedding backfill (`npm run backfill-embeddings`) — **14.9%** done (9,865 / 66,211) +- [ ] Re-measure after backfill + L0 traffic (`layer = 0` rows) + +## Post-ship telemetry (2026-08-15) + +Pulled from Neon `scan_attempts` + `cards.embedding` coverage. Single user +(`user_id = 3`); treat post-ship windows as directional, not statistically +significant. + +### All-time (n=250, 2026-05-27 → 2026-08-15) + +| Signal | Value | Baseline (2026-08-14) | Target | +| --- | --- | --- | --- | +| L1 auto-match | **5.4%** of L1 (6/111) | 5.8% | ≥25% | +| L1 escalate | **78.4%** of L1 (87/111) | 76.7% | ≤40% | +| L2 `not_a_card` | **43.9%** of L2 (61/139) | 44.6% | ≤15% | +| End-to-end auto-match | **13.2%** excl. not_a_card (25/189) | 13.5% | ≥50% | +| L1 p50 latency | **158 ms** | 158 ms | — | +| L2 p50 / p90 | **1642 / 2297 ms** | 1655 / 2349 ms | ≤1800 p50 | +| L0 attempts | **0** | — | (Phase 3 code live; index sparse) | +| `card_submissions` pending | **31** | 27 | ops | + +### Aug 15 post-deploy session (n=26) + +First scan burst after #156–#158 merged (~06:23–07:43 UTC). No L0 rows yet +(#160 landed same evening; backfill incomplete). + +| Signal | Aug 15 (n=26) | Baseline | +| --- | --- | --- | +| L1 auto-match | **0%** (0/8 L1) | 5.8% | +| L1 escalate | **100%** (8/8 L1) | 76.7% | +| L2 `not_a_card` | **38.9%** (7/18 L2) | 44.6% | +| End-to-end auto-match | **10.5%** excl. not_a_card (2/19) | 13.5% | + +**Qualitative notes:** L1 OCR strips post-warp are *shorter* garbage (`"7"`, +`"Tey"`, `"yr'"`) — collector-number path cannot help. L2 structured JSON +(`isCard: false`) works. Two L2 auto-matches (Hunger Hawk, Hog-Monkey). +Disambiguation still common (Seel printings). + +### Phase verdict + +| Phase | PR | Code | Success metrics | +| --- | --- | --- | --- | +| 1 `tighten-scan-identify-hot-path` | #156 | **Shipped** | ❌ L1 match/escalate targets missed | +| 2 `improve-scan-card-detection` | #158 | **Shipped** | ❌ not_a_card still ~44%; slight Aug 15 improvement | +| 3 `scan-visual-catalog-search` | #160, #162 | **Shipped** | ⏳ Awaiting backfill + L0 traffic | + +**Recommendation:** Finish catalog embedding backfill, scan 50+ cards on +preview, then re-query `layer = 0` distribution before tuning thresholds. + +### Embedding backfill status (2026-08-15) + +| Metric | Value | +| --- | --- | +| Cards with `image_url` | 66,211 | +| Rows with `embedding` | 9,865 (**14.9%**) | +| pgvector on Neon | ✅ v0.8.0 | +| Operator command | `npm run backfill-embeddings` | ## Worktree diff --git a/.convoys/tighten-scan-identify-hot-path.md b/.convoys/tighten-scan-identify-hot-path.md index bcaeea2..42cf3e1 100644 --- a/.convoys/tighten-scan-identify-hot-path.md +++ b/.convoys/tighten-scan-identify-hot-path.md @@ -10,7 +10,7 @@ skip: - ui-design - visual - flag -status: in-progress +status: shipped created: 2026-08-14 depends_on: - add-real-ocr-layer @@ -115,7 +115,19 @@ is never read on L1. - [x] Brief C — `scan-vision.js` JSON schema + optional model id - [x] Brief D — skip automatic disambiguation Gemini refine - [x] SCHEMA_MAP layer-1 wording -- [ ] Re-query `scan_attempts` after preview traffic +- [x] Re-query `scan_attempts` after preview traffic (2026-08-15 — metrics unvalidated; see umbrella § Post-ship telemetry) + +## Post-ship (PR #156, 2026-08-14) + +**Shipped:** faster verify gates, collector-number L1, structured Gemini JSON, +skip L1 disambiguation refine. + +**Telemetry (n=250 all-time):** L1 matched 5.4% (target ≥25%); L1 escalate +78.4% (target ≤40%). Gate latency goal met (~158 ms L1 p50). Aug 15 session +(n=26) showed 0% L1 match — OCR strips got noisier post-warp, not better. + +**Verdict:** Machinery shipped; L1 text path remains insufficient for the +umbrella 50% auto-match goal. Phase 3 visual kNN is the intended fix. ## Likely file ownership (Architect will lock)