rstillwell/deckhearth
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions 10
deckhearth/.convoys/scan-visual-catalog-search.md
varutasu 7007eae3ba
fix(ci): Phase 3 post-merge CI gates (#162) 
* fix(ci): allow card-embed.js and skip pgvector on non-superuser CI

Add lib/card-embed.js to the server-only LLM allowlist (forbidden-patterns
Check 3). Make the pgvector migration degrade gracefully when CT 102 CI
cannot CREATE EXTENSION vector so migrate up still passes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(convoy): post-ship metrics and operator checklist for Phase 3

Record Neon migration applied, scan_attempts snapshot, and backfill gate.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-14 21:53:34 -05:00

7.3 KiB
Raw Blame History

name classification success_metric skip status created depends_on umbrella model_policy
scan-visual-catalog-search feature After catalog embeddings exist and the new kNN path is on, ≥50% of legitimate card scans (scan_attempts excluding not_a_card) auto-match a printing with no picker — the original add-real-ocr-layer 70% target, measured honestly. Gemini L2 share of attempts falls vs the Phase 1 post-ship baseline.
ia
ui-design
visual
a11y
design
flag
shipped 2026-08-14
improve-scan-card-detection
scanner-identify-upgrade
default_session roles escalate_to escalate_to_premium never_premium
auto
role-conductor role-architect role-ia-architect role-ux-reviewer role-ui-designer role-implementer role-reviewer role-security-auditor role-design-system-auditor role-a11y-auditor role-doc-writer
composer-2.5-fast composer-2.5 composer-2.5-fast composer-2.5-fast composer-2.5-fast composer-2.5-fast cursor-grok-4.5-high gpt-5.6-terra-medium cursor-grok-4.5-high cursor-grok-4.5-high auto
claude-sonnet-5-thinking-medium claude-4.6-opus-high-thinking
role-reviewer
role-security-auditor
role-design-system-auditor
role-a11y-auditor
role-ui-designer
role-doc-writer

Convoy: scan-visual-catalog-search

Phase 3 of scanner-identify-upgrade. Identify cards by what they look like, not by reading the name. This is how Manabox / Delver-style scanners get printing-accurate matches on foil and alt-art.

Blocked on improve-scan-card-detection: embeddings of unwarped phone crops will not match catalog image_urls reliably.

Why

Name-only pg_trgm cannot distinguish printings. Tesseract fails on foil and stylized type even when it emits 8–20 characters (76.7% L1 escalate with non-empty text). Gemini can read a name but still returns a picker when set/number are missing, and costs 5/min.

The catalog already stores cards.image_url for every imported printing. Precompute an embedding per row; at scan time embed the warped crop and take top-k cosine. OCR / Gemini become hints and unknown-card fallback.

Scope

In scope

  • Migration: pgvector (confirm Neon availability) + cards.embedding (or a side table card_embeddings) + ANN index. Update docs/SCHEMA_MAP.md.
  • Offline embed job: new dated script or npm run task that reads image_url, writes vectors. Idempotent. Rate-limit the embedding provider. Do not edit historical scripts/add-*.js.
  • Architect picks the embedder (one):
    1. Gateway embedding model (same AI_GATEWAY_API_KEY, server-only).
    2. In-browser MobileCLIP-S2 / SigLIP ONNX for the query crop, with catalog vectors baked or fetched — only if weight license + size are acceptable.
  • Identify path: new Layer-0 (or replace L1) — kNN then auto-match / disambiguate / escalate to existing L2. Log scan_attempts.layer = 0 (or Architect-ratified value).
  • Similarity thresholds analogous to 0.85 / 0.60, tuned on a held-out set of scan captures if any exist in Blob.
  • Auth + rate-limit on any new route. No client-side API keys (forbidden-client-side-llm-keys must stay green).

Out of scope

  • Replacing Gemini entirely on day one — keep L2 for catalog misses and low similarity.
  • Training a custom card CNN.
  • Python GPU service.
  • Changing scanner chrome / cart.
  • Auto-approving the 27 pending card_submissions.

Roles invoked

  1. role-architect — embedder, schema, layer numbering, thresholds, brief split (migration / backfill / route). Security-sensitive: escalate to Sonnet if schema + new route land together.
  2. role-implementer.
  3. Audit: role-reviewer + role-security-auditor (required).

Todos

  • Architect: confirm pgvector on prod Neon tier
  • Brief 1 — migration + SCHEMA_MAP
  • Brief 2 — catalog backfill job (idempotent)
  • Brief 3 — identify kNN route + client escalate order
  • Threshold bake-off on real crops
  • Re-measure auto-match % excluding not_a_card
  • Operator: AI_GATEWAY_API_KEY + npm run backfill-embeddings on Neon (migrate applied 2026-08-15)

Post-ship (2026-08-15)

CI fixes: PR #162 — lib/card-embed.js allowlist + pgvector migration graceful skip on non-superuser homelab CI.

Neon operator: 1782000000001_add-card-embeddings applied manually (pgvector + columns + index + pgmigrations row). Full backfill pending AI_GATEWAY_API_KEY in operator env.

scan_attempts snapshot (90d, n=250, pre-backfill / no L0 traffic yet):

Signal Value Baseline
L1 escalate 78.4% (87/111) 76.7%
L1 matched 5.4% (6/111) 5.8%
L2 not_a_card 43.9% (61/139) 44.6%
End-to-end auto-match 10.0% (25/250) 10.3%

Re-measure after backfill + preview scanning for L0 layer=0 rows.

Likely file ownership

Area Files
Schema migrations/*_card-embeddings.js, docs/SCHEMA_MAP.md
Backfill new scripts/ job (dated) or lib/card-embed-backfill.js
Query new lib/card-visual-match.js, pages/api/scan/identify-by-image.js or fold into existing identify
Client lib/scanner-card-identify.js (tryLayer1TextIdentify sibling)

Do not rewrite lib/scanner-card-detection.js here.

Multitask dispatch

Brief 1 first. Brief 2 after 1. Brief 3 after 1 (can overlap 2 if the route degrades to escalate-when-empty-index).

/multitask role-reviewer + role-security-auditor

Group id: audit-scan-visual-catalog-search-<pr>.

CI impact

Workflow / job Behavior
schema-map-fresh Fires — migration + SCHEMA_MAP
ci.yml migrate Must apply pgvector on CT 102 CI Postgres — Architect must verify the extension is available there or gate the migration
forbidden-client-side-llm-keys Blocking

Operator action

  • Confirm Neon pgvector (or Neon’s equivalent) on the prod project.
  • Budget: one embedding per catalog image, plus one per live scan if the query embed is server-side. Architect publishes a cost note before Brief 2 runs against prod images.
  • No new browser secrets.

Conductor notes

This is the accuracy leap. Do not start it to "try CLIP" before Phase 2 crops are rectified — that wastes the backfill. If Architect finds pgvector unavailable on CI Postgres, stop and write a fallback (external index vs skip-CI-extension plan) rather than shipping an untestable migration.

UX

No new screens. Scan flow stays L0 → L1 → L2 with the same disambiguation picker and error toasts. When the catalog index is empty (backfill not run), L0 escalates silently with no embed cost.

Gallery uploads now try visual match before OCR.

Architecture

Decision D1 — Gateway multimodal embedder (cohere/embed-v4.0)

Server-only via AI_GATEWAY_API_KEY. 1024-dim vectors in cards.embedding. Env: SCAN_EMBED_MODEL, SCAN_EMBED_DIMENSION.

Decision D2 — Layer numbering

Layer Path
0 POST /api/scan/identify-by-image
1 Tesseract + identify-by-text
2 Gemini + scan/identify

Decision D3 — Thresholds

Match ≥ 0.82 (0.06 gap). Disambiguation ≥ 0.58.

Decision D4 — Rate limit

checkScanRateLimit on identify-by-image. L0 429 falls through to L1 (not a hard stop).

Audit group id: audit-scan-visual-catalog-search-<pr>.