deckhearth/.convoys/add-rate-limiting.md

1036 lines
55 KiB
Markdown
Raw Normal View History

---
name: add-rate-limiting
classification: convoy
success_metric: |
All endpoints listed in P0 #6 of `.convoys/ship-readiness.md` carry
a rate-limit gate appropriate to their abuse class (search /
upload / import / admin). The card-import endpoints are no longer
anonymous-callable. `lib/rate-limit.js` exposes named limiters
per route class with documented limits + windows + key-extraction
strategies. P0 #6 flips PARTIAL → RESOLVED, closing the last
open P0 ship-blocker.
skip:
- role-design-system-auditor
- role-a11y-auditor
- role-ux-reviewer
- role-ia-architect
docs: post-convoy cleanup for add-rate-limiting — MILESTONE, last P0 closed Reflects the merged add-rate-limiting convoy (PR #20, squash commit 708ef45) in repo documentation. **This is the milestone cleanup** — add-rate-limiting closed P0 #6 (No rate limiting anywhere), the LAST open P0 ship-blocker. `.convoys/ship-readiness.md`'s § Status summary flips from "7 of 8 RESOLVED; 1 remains" to **"8 of 8 RESOLVED. Launch-readiness P0 checklist is empty."** One brief in the convoy: Brief 1 shipped as planned with no scope expansions and no implementer deviations from the verbatim spec; all six architect decisions ratified verbatim at gate 1 (D1 operator-ratified Option A; D2-D6 architect-self-ratified). .convoys/add-rate-limiting.md: - frontmatter status: in-progress -> shipped (added shipped: 2026-05-24) - new ## As-shipped section. Opens with the milestone language pointing back at ship-readiness.md's flipped § Status summary. Decisions section captures all 6 ratifications (D1 operator- ratified Option A — Critical: WHY the atomic admin UI fix in pages/admin/card-import.js was Decision 1's hidden coupling requirement, since API gating alone would have broken every "Import Cards" click; D2 hybrid named-limiter shape with Map<className, Ratelimit> cache; D3 per-class table including the two D3 tuning-evidence raises — search 30 -> 60/min because ShareModal.handleSearch has no debounce so a 17-char email = 16 requests in <5s, and generate kept at 5/hour because DiceBear is free not paid AI; D4 two-extractor shape with defensive THROW on null/empty userId; D5 uniform 429 message; D6 no new vitest or playwright specs deferred to fill-vitest-handler-coverage). As-shipped surface broken into 4 layers (1 lib refactor + 6 route gates + 1 atomic admin UI fix + 1 rule extension) mirroring the cors-tighten cleanup's pattern-split shape. Empirical CI metrics from post-merge run 26382185019 (Playwright smoke 59s 3/3 in 3.8s, forbidden-cors-headers pass, vitest 21/21, lint 128 baseline, Screenshot diff continue-on-error swallow per Decision 4). Cross-validation finding: smoke test 2 still passes against the post-rate-limit preview — that's three convoys in a row (PR #15 Layout default-user, PR #19 CORS-tighten, PR #20 rate-limiting) where the same 3-test smoke spec defended the auth surface through sweeping changes. Operator-action-required: none. What did NOT change audit trail. .convoys/ship-readiness.md: - § Status summary at the top flipped from 7/8 to 8/8 RESOLVED. Header text updated to "Launch-readiness P0 checklist is empty." P0 #6 row in the table flips from PARTIAL to RESOLVED with the two-convoy lineage (fix-auth-bypass Brief 4 + add-rate-limiting). Trailing paragraph rewritten as a milestone note: security gate closed; remaining launch work is P1 quality bar + P2/P3 polish. - P0 #6 entry flipped from PARTIAL to RESOLVED with the full add-rate-limiting as-shipped block (8 sub-bullets covering the lib refactor shape, the per-class table, the defensive THROW, the three import routes' auth-gating, the atomic admin UI fix and WHY, the rule extension, the 6 decisions, and the diff breakdown). Brief 4's 2026-05-23 partial is preserved as the prior as-shipped layer to maintain the audit trail. - Launch sequence step 4 marked RESOLVED 2026-05-24 with the squash commit + smoke metrics inline. - Queued convoys: removed the add-rate-limiting entry (it shipped). Added a new delete-dead-lorcana-import entry (P3 polish; the Lorcana import route was gated defensively in PR #20 despite zero current frontend callers — pages/admin/card-import.js's <select> only offers mtg + pokemon — so if Lorcana stays permanently out of the admin UI, this is the cleanup PR). Added three "flagged but kept out of scope" follow-ups per the convoy's § What did NOT change: harden-multipart-parser (P2; 5MB body still consumed before the 429 path on avatar.js), god-function-split / refactor-cards-search-sql (P2; 240-line 7-branch SQL in cards/search.js), and withAdmin(handler) wrapper extraction (P3 DX; the three import routes are call sites #3-5 in the codebase but uniform inline shape was preserved for convoy atomicity). Updated tighten-visual-diff-path-filter to note PR #20 also tripped the same false-positive. AGENTS.md: - Gotcha #12 extended end-to-end. Was the single-class auth-only lib + the env-var contract; is now the 5-class reality with a full per-class table (helper / limit-window / key / routes), the defensive THROW pattern in extractUserIdentifier, the gate-ordering rule for per-user limiters, and the auth → admin-role → rate-limit ordering for the three import routes. Prominent milestone line opens the new content: "add-rate-limiting convoy (squash 708ef45, PR #20, 2026-05-24) closed P0 #6 — all 8 P0s now RESOLVED." Original env-var contract paragraph (KV_REST_API_URL / KV_REST_API_TOKEN, fail-closed-in-prod / warn-and-noop-in-dev) is preserved verbatim above the new content. - § 6 Testing: intentionally untouched (no test surface changed; vitest 21/21 and smoke 3/3 still apply). - § 7 Deployment: intentionally untouched (no deployment-shape changed; same KV_REST_API_* env vars from Brief 4). .cursor/rules/api-routes.mdc: - The implementer extended § Rate limiting in PR #20 with the per-class table + verbatim call shape + gate-ordering rules + identifier-extraction + uniform 429 + fail-closed env-var contract + fail-open Upstash-outage behavior. Doc-writer pass verified completeness; added a one-sentence convoy-attribution line at the top of § Rate limiting citing the two-convoy lineage (fix-auth-bypass Brief 4 for the auth class + add-rate-limiting for the other four classes and 7 newly-gated routes), mirroring the post-cors-tighten § CORS attribution shape. No other touch-ups needed. No changes to: package.json, package-lock.json, lib/rate-limit.js, pages/**, components/**, scripts/**, test/**, tests/**, .github/workflows/**, README.md, TESTING_GUIDE.md, playwright.config.js, eslint.config.mjs. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 00:09:07 -04:00
status: shipped
created: 2026-05-24
docs: post-convoy cleanup for add-rate-limiting — MILESTONE, last P0 closed Reflects the merged add-rate-limiting convoy (PR #20, squash commit 708ef45) in repo documentation. **This is the milestone cleanup** — add-rate-limiting closed P0 #6 (No rate limiting anywhere), the LAST open P0 ship-blocker. `.convoys/ship-readiness.md`'s § Status summary flips from "7 of 8 RESOLVED; 1 remains" to **"8 of 8 RESOLVED. Launch-readiness P0 checklist is empty."** One brief in the convoy: Brief 1 shipped as planned with no scope expansions and no implementer deviations from the verbatim spec; all six architect decisions ratified verbatim at gate 1 (D1 operator-ratified Option A; D2-D6 architect-self-ratified). .convoys/add-rate-limiting.md: - frontmatter status: in-progress -> shipped (added shipped: 2026-05-24) - new ## As-shipped section. Opens with the milestone language pointing back at ship-readiness.md's flipped § Status summary. Decisions section captures all 6 ratifications (D1 operator- ratified Option A — Critical: WHY the atomic admin UI fix in pages/admin/card-import.js was Decision 1's hidden coupling requirement, since API gating alone would have broken every "Import Cards" click; D2 hybrid named-limiter shape with Map<className, Ratelimit> cache; D3 per-class table including the two D3 tuning-evidence raises — search 30 -> 60/min because ShareModal.handleSearch has no debounce so a 17-char email = 16 requests in <5s, and generate kept at 5/hour because DiceBear is free not paid AI; D4 two-extractor shape with defensive THROW on null/empty userId; D5 uniform 429 message; D6 no new vitest or playwright specs deferred to fill-vitest-handler-coverage). As-shipped surface broken into 4 layers (1 lib refactor + 6 route gates + 1 atomic admin UI fix + 1 rule extension) mirroring the cors-tighten cleanup's pattern-split shape. Empirical CI metrics from post-merge run 26382185019 (Playwright smoke 59s 3/3 in 3.8s, forbidden-cors-headers pass, vitest 21/21, lint 128 baseline, Screenshot diff continue-on-error swallow per Decision 4). Cross-validation finding: smoke test 2 still passes against the post-rate-limit preview — that's three convoys in a row (PR #15 Layout default-user, PR #19 CORS-tighten, PR #20 rate-limiting) where the same 3-test smoke spec defended the auth surface through sweeping changes. Operator-action-required: none. What did NOT change audit trail. .convoys/ship-readiness.md: - § Status summary at the top flipped from 7/8 to 8/8 RESOLVED. Header text updated to "Launch-readiness P0 checklist is empty." P0 #6 row in the table flips from PARTIAL to RESOLVED with the two-convoy lineage (fix-auth-bypass Brief 4 + add-rate-limiting). Trailing paragraph rewritten as a milestone note: security gate closed; remaining launch work is P1 quality bar + P2/P3 polish. - P0 #6 entry flipped from PARTIAL to RESOLVED with the full add-rate-limiting as-shipped block (8 sub-bullets covering the lib refactor shape, the per-class table, the defensive THROW, the three import routes' auth-gating, the atomic admin UI fix and WHY, the rule extension, the 6 decisions, and the diff breakdown). Brief 4's 2026-05-23 partial is preserved as the prior as-shipped layer to maintain the audit trail. - Launch sequence step 4 marked RESOLVED 2026-05-24 with the squash commit + smoke metrics inline. - Queued convoys: removed the add-rate-limiting entry (it shipped). Added a new delete-dead-lorcana-import entry (P3 polish; the Lorcana import route was gated defensively in PR #20 despite zero current frontend callers — pages/admin/card-import.js's <select> only offers mtg + pokemon — so if Lorcana stays permanently out of the admin UI, this is the cleanup PR). Added three "flagged but kept out of scope" follow-ups per the convoy's § What did NOT change: harden-multipart-parser (P2; 5MB body still consumed before the 429 path on avatar.js), god-function-split / refactor-cards-search-sql (P2; 240-line 7-branch SQL in cards/search.js), and withAdmin(handler) wrapper extraction (P3 DX; the three import routes are call sites #3-5 in the codebase but uniform inline shape was preserved for convoy atomicity). Updated tighten-visual-diff-path-filter to note PR #20 also tripped the same false-positive. AGENTS.md: - Gotcha #12 extended end-to-end. Was the single-class auth-only lib + the env-var contract; is now the 5-class reality with a full per-class table (helper / limit-window / key / routes), the defensive THROW pattern in extractUserIdentifier, the gate-ordering rule for per-user limiters, and the auth → admin-role → rate-limit ordering for the three import routes. Prominent milestone line opens the new content: "add-rate-limiting convoy (squash 708ef45, PR #20, 2026-05-24) closed P0 #6 — all 8 P0s now RESOLVED." Original env-var contract paragraph (KV_REST_API_URL / KV_REST_API_TOKEN, fail-closed-in-prod / warn-and-noop-in-dev) is preserved verbatim above the new content. - § 6 Testing: intentionally untouched (no test surface changed; vitest 21/21 and smoke 3/3 still apply). - § 7 Deployment: intentionally untouched (no deployment-shape changed; same KV_REST_API_* env vars from Brief 4). .cursor/rules/api-routes.mdc: - The implementer extended § Rate limiting in PR #20 with the per-class table + verbatim call shape + gate-ordering rules + identifier-extraction + uniform 429 + fail-closed env-var contract + fail-open Upstash-outage behavior. Doc-writer pass verified completeness; added a one-sentence convoy-attribution line at the top of § Rate limiting citing the two-convoy lineage (fix-auth-bypass Brief 4 for the auth class + add-rate-limiting for the other four classes and 7 newly-gated routes), mirroring the post-cors-tighten § CORS attribution shape. No other touch-ups needed. No changes to: package.json, package-lock.json, lib/rate-limit.js, pages/**, components/**, scripts/**, test/**, tests/**, .github/workflows/**, README.md, TESTING_GUIDE.md, playwright.config.js, eslint.config.mjs. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 00:09:07 -04:00
shipped: 2026-05-24
parent: ship-readiness
addresses: P0 #6 (PARTIAL → RESOLVED) — the LAST open P0
depends_on:
- fix-auth-bypass (Brief 4 shipped lib/rate-limit.js + auth-only limiter)
- cors-tighten (just closed P0 #5; not a hard dep but the API surface is now CORS-clean)
---
# Convoy: add-rate-limiting
Extend `lib/rate-limit.js` and wire the result into the remaining
abusable endpoints. This is the final P0 — when it lands, the
launch-readiness ship-blocker list is empty (all 8 of 8 RESOLVED).
## Why now
`fix-auth-bypass` Brief 4 shipped the rate-limit infrastructure
(`lib/rate-limit.js`, `@upstash/ratelimit@^2.0.8`,
`@upstash/redis@^1.38.0`, Vercel-Upstash Marketplace env vars
`KV_REST_API_URL` / `KV_REST_API_TOKEN`) and wired it into the
two auth endpoints (`login`, `register`) with a 5-attempts /
15-minute sliding window. The remaining abusable surface was
intentionally deferred to this convoy.
The "Queued convoys" entry in `.convoys/ship-readiness.md` lists
the surface as:
- `/api/users/search`
- `/api/cards/search`
- All `/api/cards/import-*`
- `/api/user/avatar*` (upload)
Parent's pre-architect audit at convoy creation surfaced a
**critical secondary finding** beyond "missing rate limit":
**`pages/api/cards/import-mtg.js`, `import-pokemon.js`,
`import-lorcana.js` have ZERO authentication checks.** They are
publicly callable, they hit Scryfall / Pokémon-TCG / Lorcana APIs
with no caller throttling AND no auth, and they perform UPSERTs
into the `cards` table. An attacker can:
1. Trigger expensive external API calls from your IP (Scryfall +
Pokémon-TCG have published rate limits; you'd hit them with no
recourse, getting your IP throttled at the source).
2. Cause unbounded DB writes (each import inserts hundreds-to-thousands
of rows; `card.id` lookups deduplicate but the INSERT path runs
in a tight per-card loop).
3. Even with a generous per-IP rate limit (5 req / hour), with
1000 IPs the math is 120,000 imports / day — enough to exhaust
Vercel function quotas and Neon row budgets.
The rate-limit alone is insufficient. **Auth-gate-then-rate-limit
is the correct shape.** Whether to add auth gates to the import
routes in this convoy (vs. spinning out a separate
`gate-import-routes` convoy) is **architect Decision 1** below.
Parent recommends in-scope; expansion is mid-convoy precedent
established by `drop-public-setup` (Brief 2's CJS/ESM expansion)
and other prior convoys.
## Scope
### Confirmed in-scope (regardless of architect Decision 1)
- **`lib/rate-limit.js`** — refactor to expose multiple named
limiters (one per route class), preserving the existing
`checkAuthRateLimit` export for `login` / `register` backwards
compatibility. New named limiters per Decision 2 below.
- **`pages/api/users/search.js`** — add rate-limit gate after the
existing JWT verification. Limit per Decision 2.
- **`pages/api/cards/search.js`** — add rate-limit gate at top of
handler (this endpoint is anonymous-by-design; rate-limit keyed
by IP).
- **`pages/api/user/avatar.js`** — add rate-limit gate after the
existing `getUserFromRequest` check. Limit keyed by `user.userId`
(authenticated; per-user makes more sense than per-IP for an
upload endpoint where a household might share an IP).
- **`pages/api/user/avatar/generate.js`** — same shape as
`avatar.js`. Per-user, stricter than avatar.js (AI generation
is more expensive than blob upload).
- **`.cursor/rules/api-routes.mdc`** § Rate limiting — update the
existing § with the new per-class pattern + each named limiter's
use case. The current § shows only the auth pattern.
### Scope-expansion-candidate (architect Decision 1)
- **`pages/api/cards/import-mtg.js`** — add `getUserFromRequest`
check at top of handler; return 403 if not admin (`user.role !==
'admin'`); then rate-limit gate (per-user, very strict — these
hit external APIs).
- **`pages/api/cards/import-pokemon.js`** — same shape.
- **`pages/api/cards/import-lorcana.js`** — same shape.
- (If Decision 1 = Option A "in scope": ship in this PR. If Option
B "spin out": queue `gate-import-routes` follow-up convoy and
mark P0 #6 RESOLVED-with-caveat at convoy close.)
### Out of scope
- **Global IP-based backstop limiter** (e.g., 1000 req / min per
IP across all routes via Next.js middleware). Useful but
separate scope; `add-global-rate-limit-middleware` would be its
own convoy.
- **Rate-limit headers on success responses**
(`X-RateLimit-Remaining`, `X-RateLimit-Reset`). The existing
`checkAuthRateLimit` returns `remaining` + `reset` in its
result object but `login.js` doesn't propagate them on
success; just on 429. Honoring the existing convention.
- **`@upstash/ratelimit` version bump** — pin stays at `^2.0.8`
from Brief 4. Bumping is its own convoy.
- **Per-route handler unit tests** (still deferred to the
queued `fill-vitest-handler-coverage` convoy — same reasoning
as `cors-tighten`).
- **Refactoring the existing `cards/search.js` SQL** (the file
has a known god-function shape with 7+ conditional SQL
branches; that's `god-function-split` scope, not here).
- **Verifying the avatar `parseMultipartFormData` body parser**
is rate-limit-safe (the body is read in `req.on('data')` before
any rate-limit gate could short-circuit; an attacker can still
exhaust the 5MB body even on a 429 path). That's a separate
body-streaming-defense concern (`harden-multipart-parser`) and
not in scope here.
## Operator action required
**None.** All Upstash env vars (`KV_REST_API_URL` /
`KV_REST_API_TOKEN`) are already auto-provisioned via the Vercel
Marketplace integration (seeded for `fix-auth-bypass` Brief 4).
No new secrets, no new dependencies (the `@upstash/ratelimit` +
`@upstash/redis` packages are already installed).
## Decisions to ratify with operator (architect routes)
1. **Scope expansion — auth gates on `pages/api/cards/import-*.js`
in this convoy?**
- **Option A — In scope (parent recommends).** Add
`getUserFromRequest` + admin-role check + rate-limit gate to
all 3 import routes in this PR. ~30 LOC across 3 files.
Closes the full P0 #6 attack surface in one shot. Precedent:
`drop-public-setup` Brief 2 (CJS/ESM expansion).
- **Option B — Spin out.** Stay narrow on the 4 originally-listed
routes; queue `gate-import-routes` as a separate convoy. Mark
P0 #6 RESOLVED-with-caveat noting the import-route auth gap.
- **Option C — Hybrid.** Add the rate-limit gate to import
routes now (with a clear `TODO: requires-auth` comment),
gate auth later. Worst of both worlds — leaves an
anonymous-callable abusive endpoint live with only IP-based
throttling.
- Architect investigates: confirm the import routes are
intended to be admin-only (per AGENTS.md / scripts/ folder
conventions), confirm Decision-A scope size is bounded, route
back recommendation.
2. **Named-limiter shape in `lib/rate-limit.js`.** The existing
`checkAuthRateLimit(req)` function has the limit + window
hardcoded. To support per-class limits cleanly, two main
patterns:
- **(a) Named-export functions.** Each route class gets its
own exported function: `checkSearchRateLimit`,
`checkUploadRateLimit`, `checkImportRateLimit`. Internal
`init()` builds one `Ratelimit` instance per class, cached
under different Redis prefixes. Verbose but very explicit at
each call site.
- **(b) Generic `checkRateLimit(req, options)`.** Single
exported function takes a config object. Calling code
becomes `await checkRateLimit(req, { class: 'search' })` or
similar. Less verbose at the lib but more at each call site,
and the per-route limits are documented in the lib module
rather than implicit in the function name.
- **(c) Hybrid.** Keep `checkAuthRateLimit` (used by login +
register; high-stakes contract; don't break). Add named
functions for the other 3 classes (search, upload, import).
Best of both — preserves Brief 4's contract, names new
classes explicitly.
- Parent recommends (c). Architect ratifies after reading
`lib/rate-limit.js` shape.
3. **Per-class limit values.** Reference table (architect's
to tune):
- **`auth`** (existing) — 5 / 15min, IP-keyed. (Don't change.)
- **`search`** (new) — recommend 30 / 1min, IP-keyed.
Legitimate users type-ahead-search; 30/min is generous but
stops a scraper in its tracks within 2-3s. Same IP-keying
as auth.
- **`upload`** (new — avatar.js, avatar/generate.js) —
recommend 10 / 1hour, user-keyed. Avatar uploads are rare;
10/hour catches accidental loops without blocking a user
who hits "save" three times by accident. User-keyed because
the user is authenticated and a household IP shouldn't
punish other users.
- **`generate`** (new — avatar/generate.js specifically) —
recommend 3 / 1hour, user-keyed. AI generation costs money;
stricter than plain upload.
- **`import`** (new — cards/import-*.js) — recommend 5 / 1hour,
user-keyed (if Decision 1 = Option A). Admin-triggered, hits
external APIs with their own rate limits; 5/hour is plenty.
4. **Identifier extraction.** `lib/rate-limit.js`'s
`extractIdentifier()` currently always extracts the first
`x-forwarded-for` IP. For per-user limiters (upload, generate,
import), we need a `user_id`-based extractor. Three options:
- **(a) Two extractors.** `extractIpIdentifier` (existing) +
`extractUserIdentifier(req, userId)` (new). Each limiter
calls the right one.
- **(b) Single extractor takes an `opts.user` flag.** Less
code but harder to read.
- **(c) Caller passes `key` directly.** Each handler calls
`checkUploadRateLimit(req, { key: user.userId })` and the lib
just trusts it.
- Parent recommends (a) — explicit, two small functions,
hard to misuse.
5. **429 response shape.** Match the existing
`login.js`/`register.js` pattern verbatim:
```js
res.setHeader('Retry-After', Math.ceil((reset - Date.now()) / 1000));
return res.status(429).json({ error: 'Too many attempts. Try again later.' });
```
Architect: should the error message vary per class (e.g., "Too
many search requests" vs "Too many uploads") or stay uniform?
Recommend uniform — minimizes attacker-fingerprinting of which
routes have which limits. Architect ratifies.
6. **Test coverage in this convoy.** No new per-route handler
tests (per Decision-4-equivalent from `cors-tighten`; same
deferred-to-`fill-vitest-handler-coverage` reasoning). The
Playwright smoke spec doesn't exercise any of these routes,
so no smoke coverage either. **The fail-loud-in-prod
semantics in `lib/rate-limit.js`'s `init()` function are the
integration test** — if the env vars are unset on a deployed
env, every limit-gated route fails closed on first call.
## Known constraints
- **`lib/rate-limit.js` already has the right architecture** for
multiple limiters (lazy `init()` returns a `cached` object;
refactor pattern is to make `cached` a Map<className,
Ratelimit> instead of a single instance). Don't rewrite from
scratch; extend.
- **Fail-closed semantics in prod must be preserved.** Brief 4's
`init()` throws when env vars are unset and `NODE_ENV ===
'production'`. New limiters must inherit this — a misconfigured
prod env should fail loud on every call, not silently disable
all rate limiting.
- **Card-search is anonymous-by-design** — do NOT add a
`getUserFromRequest` check. The IP-keyed limit is the correct
defense (search is a public catalogue feature).
- **User-keyed limits require an authenticated request.** The
rate-limit gate MUST sit AFTER the auth check in the handler
body. Wrong order = security regression (anonymous user bypasses
the gate because there's no `userId` to key on, the extractor
falls back to IP, and the per-user limit becomes per-IP — same
household IP could be locked out for a different user's
behavior).
- **Brief 4's `lib/rate-limit.js` is unit-tested transitively**
via the existing vitest suite (none of the 21 tests
specifically test rate-limit, but the auth-utils tests run
through the same module). Don't break those.
- **`@upstash/ratelimit` supports multiple shapes**
(`slidingWindow`, `fixedWindow`, `tokenBucket`); Brief 4 used
`slidingWindow`. New limiters should follow the same shape
unless the architect has a specific reason for a different
algorithm per class.
- **Per-route Pattern checks** — the 6 (or 9 if Decision 1 =
Option A) target files have different existing structures:
- `users/search.js`: auth check inside try/catch on JWT;
rate-limit goes after JWT verify, inside the existing try
- `cards/search.js`: no auth; rate-limit goes at top of
handler after the method check
- `user/avatar.js`: auth check inside the outer try (line
16-19); rate-limit goes after auth, before the method
branching at line 21
- `user/avatar/generate.js`: similar shape to avatar.js
- `cards/import-*.js`: NO auth currently. Decision 1 dictates
the shape — either add auth-then-rate-limit, or rate-limit-only
(Option C, not recommended)
## Acceptance criteria
The convoy is shippable when ALL of the following hold:
1. `lib/rate-limit.js` exposes the named limiters per Decision 2.
Backwards-compat: `checkAuthRateLimit` still works for
`login.js` / `register.js` (do NOT break Brief 4's contract).
2. All 4 originally-listed endpoints (`users/search`,
`cards/search`, `user/avatar`, `user/avatar/generate`) carry
the appropriate rate-limit gate at the correct ordering
(after auth if applicable; per Decision 4 keying).
3. If Decision 1 = Option A: all 3 import endpoints carry both
the auth gate (admin-role check) + the import rate-limit gate.
4. `npm run lint` exit code matches baseline (still 128 problems;
do NOT regress).
5. `npm run test:run` (vitest) still passes 21/21 (no regression
on the existing auth-utils tests that transitively load
`lib/rate-limit.js`).
6. `npm run test:smoke` (via CI on the PR) still passes 3/3
against the Vercel preview — proves the live login → verify
→ /api/health flow doesn't accidentally get rate-limited
(the smoke spec hits each endpoint once per run, well below
any limit).
7. `.cursor/rules/api-routes.mdc` § Rate limiting updated to
document each new limiter's use case + the per-class limits
+ the user-keyed vs IP-keyed convention.
8. Bypass-secret-leak check: zero matches of any
`KV_REST_API_*` or `UPSTASH_*` env-var values in any CI log.
(Auto-masked by GitHub Actions; verify post-merge.)
## Anything flagged but not acted on (in advance)
- **Global IP backstop limiter via Next.js middleware** —
out of scope; queue `add-global-rate-limit-middleware` if a
future audit shows non-listed routes being abused.
- **Body-streaming defense for avatar uploads** — the 5MB
multipart body is consumed before any rate-limit gate can
short-circuit. Real defense requires moving the parse into a
separate edge function or using `read-up-to` semantics; queue
`harden-multipart-parser` if it ever surfaces in a real abuse
incident.
- **Per-tier user limits** — premium users might get higher
search/upload limits. Queue `tiered-rate-limits` when a
tiering scheme exists (today there are only `admin` and
default roles).
- **Rate-limit metrics dashboard** — Upstash exposes per-key hit
counts; surfacing them in an admin dashboard would catch
abuse early. Queue `rate-limit-observability` post-launch.
- **Card-search SQL god-function** — known shape issue,
separate convoy (`god-function-split` or
`refactor-cards-search-sql`); do NOT touch in this convoy.
## Decisions (post-architect)
Six decisions ratified by `role-architect` on 2026-05-24. **Only Decision 1
requires operator gate-1 sign-off** (significant scope expansion to admin-only
import-route enforcement + admin UI source touch); the other five are
architect-self-ratifiable per the precedent established by `cors-tighten`
Decision D2-D5 + `fix-vercel-deployment-protection-in-ci` Decisions A/B/D.
### Decision 1 — Scope expansion: gate `pages/api/cards/import-*.js` in this convoy (**OPERATOR-RATIFIABLE**)
**Ratified: Option A — gate all three import routes with auth + admin-role + import rate-limit, AND fix the admin UI to send the Bearer token.**
Architect investigation results:
- **`pages/api/cards/import-mtg.js`** — LIVE admin tooling. Called by
`pages/admin/card-import.js` line 39-49 (the `<select>` defaults to
`'mtg'`). Currently fetched WITHOUT an `Authorization` header.
- **`pages/api/cards/import-pokemon.js`** — LIVE admin tooling. Same
admin UI fetch path; the `<select>` second option is `'pokemon'`.
- **`pages/api/cards/import-lorcana.js`** — DEAD in frontend. Architect
ran `rg 'import-lorcana' pages/ components/` and the file has zero
frontend callers; the admin UI's `<select>` only offers `'mtg'` and
`'pokemon'`. The standalone `scripts/import-lorcana.js` exists as an
independent CLI path. Gating defensively is the chosen path because:
(a) future Lorcana admin UI work inherits protection automatically,
(b) the diff is uniform (3 routes get identical treatment), (c) a
future cleanup convoy can delete the route if it stays unused — and
deletion is strictly easier than gating-then-deleting because Option A
preserves all the import infrastructure.
**Critical scope expansion required by Option A:**
`pages/admin/card-import.js` currently calls the import APIs without an
`Authorization` header (line 43-49). The moment the import APIs gain
`getUserFromRequest`, the admin UI begins returning 401 on every import
click — visible UX regression. The brief therefore includes a one-line
edit to `pages/admin/card-import.js` adding
`'Authorization': \`Bearer ${localStorage.getItem('auth_token')}\``
to the fetch's headers. This is the minimum touch; the 309-line
god-component is otherwise byte-identical.
**Rejected alternatives:**
- **Option B (spin out `gate-import-routes`).** Would leave a publicly-callable
abuse vector live for at least one more convoy cycle; closes P0 #6 only
partially. Rejected.
- **Option C (hybrid — rate-limit-only without auth).** Worst of both worlds:
anonymous abuse + only IP-based throttling. Rejected per the convoy spec.
- **Option D-full (delete all three import routes).** Would break the admin
UI immediately and require either deleting `pages/admin/card-import.js` too
or restructuring it to call something else. Wider scope than gating; defeats
the convoy's "close P0 #6 in one shot" success metric.
- **Option D-partial (delete only `import-lorcana.js`, gate mtg+pokemon).**
Considered. Slightly smaller attack surface, but introduces a non-uniform
pattern (two gated routes + one deleted route) that complicates the
reviewer's mental model. Lorcana support is a documented game in
`AGENTS.md` § 1; deleting the API forecloses the trivial path to a future
Lorcana admin UI. Rejected as marginal; queue `delete-dead-lorcana-import`
as a follow-up convoy if Lorcana never gets wired into the admin UI.
**Operator action**: ratify Option A or counter-propose. Architect proceeds
to brief generation assuming Option A; if operator counters before
implementer dispatch, the brief is revised in place.
### Decision 2 — Named-limiter shape in `lib/rate-limit.js` (architect-self-ratifiable)
**Ratified: Option (c) hybrid — preserve `checkAuthRateLimit(req)`, add four named functions.**
The verbatim new module shape is specified in
`.convoys/add-rate-limiting/brief-1-extend-rate-limit-and-wire-routes.md`
§ `lib/rate-limit.js` (modified). Key design choices:
- `Map<className, Ratelimit>` instance cache (one Redis client, five
Ratelimit instances, distinct prefix per class). Each `Ratelimit`
construction is a cheap object wrap around the shared Redis client; no
separate REST connection per class.
- Five exported functions: `checkAuthRateLimit(req)`,
`checkSearchRateLimit(req)`, `checkUploadRateLimit(req, userId)`,
`checkGenerateRateLimit(req, userId)`, `checkImportRateLimit(req, userId)`.
Internal `check(className, identifier)` shared helper.
- `extractIpIdentifier(req)` is module-private (renamed from the
pre-refactor `extractIdentifier` — that legacy name is gone but the
behavior is identical).
- `LIMITER_CONFIG` is a top-level const map of `{ limit, window, prefix }`
per class; adding a sixth class is a one-line addition + one new
exported function (no init() restructuring needed).
- Brief 4's `checkAuthRateLimit(req)` return shape is **byte-identical**
post-refactor (`{ allowed, remaining, reset }`) — Brief 4 contract
preserved.
### Decision 3 — Per-class limit values (architect-self-ratifiable)
**Ratified with tuning evidence:**
| Class | Limit | Window | Key | Rationale |
| --- | --- | --- | --- | --- |
| `auth` | 5 | 15 min | IP | Brief 4 unchanged. |
| `search` | **60** | 1 min | IP | **Raised from parent's 30.** `components/ShareModal.js::handleSearch` (lines 56-77) fires on every keystroke with no debounce; typing a 17-char email = 16 requests in <5s, which would 429 at 30/1min on a single legitimate user entry. 60/1min covers a realistic burst and still stops a scraper. |
| `upload` | 10 | 1 hour | user | Avatar uploads are rare; 10/hour catches accidental loops. User-keyed because a household IP shouldn't punish other users. |
| `generate` | **5** | 1 hour | user | **Raised from parent's 3.** `pages/api/user/avatar/generate.js` calls DiceBear (free public API), not OpenAI/Replicate; cost is Vercel blob + DiceBear-side throttling, not per-call $. 5/hour still catches loops without blocking a user trying 3-4 seeds. |
| `import` | 5 | 1 hour | user | Admin-only via Decision 1; hits Scryfall/Pokémon-TCG/Lorcana APIs with their own rate limits. 5/hour is plenty for the actual import workflow (one set per click; admin won't import 5 sets per hour in normal operation). |
If the operator wants different numbers, the change is a single-line edit
to `LIMITER_CONFIG` in `lib/rate-limit.js` — call out in PR review.
### Decision 4 — Identifier extraction shape (architect-self-ratifiable)
**Ratified: Option (a) — two extractors, `extractUserIdentifier` THROWS on missing userId.**
Verbatim shape in the brief. Key safety property:
`extractUserIdentifier(userId)` THROWS the message
`'[rate-limit] extractUserIdentifier called without an authenticated userId. Place the rate-limit gate AFTER the auth check, never before.'`
when `userId` is `null`, `undefined`, `''`, or `NaN`. This surfaces
gate-ordering bugs **at dev time** rather than silently falling back to
IP and converting a per-user limit into a per-IP limit — which would
lock other household members out for one user's behavior, the exact
regression flagged in `.convoys/add-rate-limiting.md` § Known constraints.
Numeric `0` is intentionally NOT in the throw conditional — if a future
schema introduces user ID 0 the limiter still keys correctly as
`'user:0'`. There is no current user with ID 0 in the `users` table; the
check is defensively forward-compatible.
### Decision 5 — 429 response shape (architect-self-ratifiable)
**Ratified: uniform message across all five classes** — `'Too many attempts. Try again later.'`.
Matches `login.js` + `register.js` verbatim. Per-class variation (e.g.,
`'Too many search requests'` vs `'Too many upload requests'`) was
considered and rejected: per-class messages would fingerprint to an
attacker which routes have which limits + windows, making it easier to
craft a request pattern that avoids 429 on the more-permissive routes
while still abusing the less-permissive ones. Single uniform message =
attacker doesn't know what they hit.
`Retry-After` calculation is also identical:
`Math.ceil((reset - Date.now()) / 1000)`. Status code is `429`.
### Decision 6 — Test coverage in this convoy (architect-self-ratifiable)
**Ratified: no new vitest or playwright tests.**
Same reasoning as `cors-tighten` Decision D4. Per-route handler tests
are deferred to the queued `fill-vitest-handler-coverage` convoy. The
fail-loud-in-prod predicate in `lib/rate-limit.js::init()` is the
integration test — if `KV_REST_API_*` is unset on a deployed env, every
gated route fails closed on first call.
**Correction to the convoy file's "Known constraints" claim:** the file
states *"Brief 4's `lib/rate-limit.js` is unit-tested transitively via
the existing vitest suite (none of the 21 tests specifically test
rate-limit, but the auth-utils tests run through the same module)."*
This is stale. Architect ran `rg 'rate-limit|@upstash' test/` → zero
matches. `test/api/auth-utils.test.js` only imports
`pages/api/auth-utils.js` + `lib/auth-secret.js` (with a mock for
`lib/database.js`); it does NOT load the auth handlers or
`lib/rate-limit.js`. The lib refactor is therefore strictly safer than
the convoy file implies — there is no transitive test path to break.
Decision 6 still holds.
## Architecture
### File plan
| File | Action | Purpose |
| --- | --- | --- |
| `lib/rate-limit.js` | modified | Refactor from single auth-only limiter into Map-of-named-limiters with five classes. Preserves Brief 4's `checkAuthRateLimit(req)` contract; adds four new named exports. |
| `pages/api/users/search.js` | modified | Add `checkSearchRateLimit(req)` gate after the inline JWT verify, before query-length validation. IP-keyed. |
| `pages/api/cards/search.js` | modified | Add `checkSearchRateLimit(req)` gate at top of handler, after method check, before the 240-line SQL god-function (which stays byte-identical). IP-keyed, anonymous-by-design. |
| `pages/api/user/avatar.js` | modified | Add `checkUploadRateLimit(req, user.userId)` after the existing `getUserFromRequest` check, before the POST/DELETE branching. User-keyed; gate fires before `parseMultipartFormData` body parse. |
| `pages/api/user/avatar/generate.js` | modified | Add `checkGenerateRateLimit(req, user.userId)` after the existing `getUserFromRequest` check, before the user-data SQL query. User-keyed; calls DiceBear (free), not a paid AI service. |
| `pages/api/cards/import-mtg.js` | modified | Add `getUserFromRequest` + `if (user.role !== 'admin') return 403` + `checkImportRateLimit(req, user.userId)` before the existing `try` block. Closes the publicly-callable anonymous-abuse vector. |
| `pages/api/cards/import-pokemon.js` | modified | Same shape as `import-mtg.js`. Live admin tooling per architect investigation. |
| `pages/api/cards/import-lorcana.js` | modified | Same shape as `import-mtg.js`. Gated defensively despite zero current frontend callers — future Lorcana admin UI inherits protection. |
| `pages/admin/card-import.js` | modified | **Scope expansion for D1.** Add `'Authorization': \`Bearer ${localStorage.getItem('auth_token')}\`` to the import fetch's headers (line 45-47). One-line addition; 309-line god-component otherwise byte-identical. |
| `.cursor/rules/api-routes.mdc` | modified | Replace § Rate limiting (lines 104-135) with the new per-class table + verbatim call shape + gate-ordering rules + identifier-extraction documentation. Every other section byte-identical. |
### API surface
No new routes. No request/response shape changes on the 7 gated routes
(429 is added to the possible status codes for each). The only public
surface change is:
- `/api/cards/import-mtg` POST: now requires `Authorization: Bearer <admin-token>`. Returns 401 (no token), 403 (non-admin), 429 (rate-limited), or the existing 200/400/500 contract.
- `/api/cards/import-pokemon` POST: same.
- `/api/cards/import-lorcana` POST: same.
- `/api/users/search` GET: adds 429 to the possible status codes (existing 401/400/200/500 unchanged).
- `/api/cards/search` GET: adds 429 to the possible status codes (existing 200/500 unchanged; no auth either pre- or post-edit).
- `/api/user/avatar` POST/DELETE: adds 429 (existing 401/400/200/500 unchanged).
- `/api/user/avatar/generate` POST: adds 429 (existing 401/404/200/500 unchanged).
All 429 responses set `Retry-After: <seconds>` and return
`{ "error": "Too many attempts. Try again later." }`.
### Schema diff
**No schema changes.** No new Postgres tables, columns, or indexes. The
limiter state lives in Upstash Redis (managed); the existing
`KV_REST_API_URL` / `KV_REST_API_TOKEN` env vars (auto-provisioned by
Vercel's Upstash Marketplace integration) cover all five classes.
### Test plan
Per Decision 6, **no new vitest or playwright specs are added in this convoy**.
Existing test coverage that must still pass post-refactor:
- `test/lib/auth-secret.test.js` (3 tests) — unrelated to rate-limit.
- `test/lib/permission-middleware.test.js` (8 tests) — unrelated.
- `test/api/auth-utils.test.js` (5 tests) — unrelated (does NOT import the auth handlers, verified at architect time).
- `test/components/Layout.test.js` (5 tests) — unrelated.
Total: 21/21 must still pass. `npm run test:run` is BLOCKING in CI per
the `test:` job in `.github/workflows/ci.yml`.
Playwright smoke (3/3, runs against Vercel preview):
- `home redirects or renders without 5xx` — anonymous GET on `/`; not in scope.
- `sign-in page renders` — anonymous GET on `/login`; not in scope.
- `public health endpoint responds` — anonymous GET on `/api/health`; not in scope.
None of the smoke tests exercise any of the 7 gated endpoints, so no
smoke regression risk. Smoke run must stay 3/3 green.
Manual verification (run locally pre-PR, paste output in PR description):
documented in the brief's § Manual verification.
### Risk list
1. **Brief 4 contract regression on `checkAuthRateLimit(req)`.** If the
refactor changes the return shape from `{ allowed, remaining, reset }`,
`pages/api/auth/login.js` + `pages/api/auth/register.js` break silently
(destructuring undefined). Mitigation: the brief locks the return shape
as byte-identical; manual verification exercises the 6th-attempt 429
path on `/api/auth/login`.
2. **Per-class Redis prefix collision.** If two `LIMITER_CONFIG` entries
accidentally share a prefix (typo, copy-paste), a search hit eats from
the auth budget for the same IP. Mitigation: brief acceptance criterion
explicitly checks `rg "tcgvault:" lib/rate-limit.js | sort -u` returns
five distinct lines.
3. **Gate-ordering reversal for user-keyed limiters.** If an implementer
places `checkUploadRateLimit(req, user.userId)` BEFORE the
`getUserFromRequest` check, every anonymous request throws (via
`extractUserIdentifier`'s null guard). Mitigation: the throw IS the
defensive signal — it surfaces as a dev-time 500 immediately rather than
a silent security regression. The brief's `## Cross-file checks` calls
out the order-check explicitly.
4. **Admin UI broken on first run if `pages/admin/card-import.js` Bearer
fix doesn't land.** Adding `getUserFromRequest` to the import APIs
without the matching admin-UI fix produces an immediate 401 on the next
"Import Cards" click. Mitigation: brief includes both edits as a single
atomic change; PR review enforces atomicity (acceptance criterion calls
the dependency out explicitly).
5. **Search 60/1min still too low for rapid typists.** If 60/min ends up
too restrictive in production, surface as a `tune-search-rate-limit`
follow-up convoy; the fix is a single LIMITER_CONFIG edit (60 → 90 or
120). Not a release-blocker — the failure mode is a 429 with a
`Retry-After: 60` header, which the front-end can display as
"searching too fast, try again in a moment".
6. **`extractUserIdentifier` throws on edge-case userIds.** Numeric `0` is
intentionally accepted (returns `'user:0'`). Empty-string userId throws.
If a future auth refactor changes the `userId` type to a UUID string,
the empty-string guard still works. If it changes to an opaque `object`,
the throw fires (correct — we don't want to key off an object).
7. **Upstash outage fails-open.** Brief 4's design choice carries through:
on `ratelimit.limit(...)` failure, return `{ allowed: true, ... }`. A
hard Upstash outage during an attack would defeat the limiter for the
duration of the outage. Mitigation: defense-in-depth (Vercel firewall,
future fail2ban-style lockout) and Upstash's published SLA. Not
addressable inside this convoy.
8. **Body-stream bypass on avatar.js.** `parseMultipartFormData` consumes
the 5MB body via `req.on('data')` before the response is sent, so an
attacker can still exhaust the 5MB ceiling per 429. This is explicitly
out-of-scope (`harden-multipart-parser` convoy). The brief's gate
ordering places the limiter BEFORE the method branches (which call
`parseMultipartFormData`), so when that future hardening lands, the
gate ordering is already correct.
### Verbatim new `lib/rate-limit.js` shape
Specified in full in
`.convoys/add-rate-limiting/brief-1-extend-rate-limit-and-wire-routes.md`
§ `lib/rate-limit.js` (modified). Implementer has zero design discretion.
## Decomposition
| Brief # | Title | Files | Depends on | Estimated PR size |
| --- | --- | --- | --- | --- |
| 1 | Extend `lib/rate-limit.js` to named per-class limiters + wire into the remaining abusable endpoints + gate the import routes | `lib/rate-limit.js`, `pages/api/users/search.js`, `pages/api/cards/search.js`, `pages/api/user/avatar.js`, `pages/api/user/avatar/generate.js`, `pages/api/cards/import-mtg.js`, `pages/api/cards/import-pokemon.js`, `pages/api/cards/import-lorcana.js`, `pages/admin/card-import.js`, `.cursor/rules/api-routes.mdc` | — | ~180 LOC across 10 files (lib refactor ~65 lines net add; 7 route edits ~6-16 lines each; admin UI +1 line; rules doc ~70 lines + / ~32 lines -) |
**Brief count: 1.** Splitting into two briefs (one for lib refactor, one
for per-route wiring) was considered and rejected:
- Per-route wiring **depends on** the lib refactor (the new exports
don't exist until Brief 1 lands), so parallelism gain via
`/multitask` is zero — Brief 2 would have `depends_on: [1]` and run
serially anyway.
- Single brief presents the reviewer with one coherent change instead
of two related-but-fragmented PRs.
- Cross-brief commitment overhead (Brief 1 ships stub limiters, Brief 2
resolves the wiring) creates exactly the kind of forward-declaration
scaffolding the `role-architect` Boot-the-brief check exists to
prevent.
The single brief is well-bounded at ~180 LOC across 10 files (mostly
small additions). No single file gets more than ~70 lines of edit; the
average file edit is ~12 lines.
### Slice dependencies (multitask-ready)
```yaml
slice_dependencies:
- brief: 1
depends_on: []
files:
- lib/rate-limit.js
- pages/api/users/search.js
- pages/api/cards/search.js
- pages/api/user/avatar.js
- pages/api/user/avatar/generate.js
- pages/api/cards/import-mtg.js
- pages/api/cards/import-pokemon.js
- pages/api/cards/import-lorcana.js
- pages/admin/card-import.js
- .cursor/rules/api-routes.mdc
```
One brief, no `/multitask` fan-out — the conductor dispatches a single
implementer.
docs: post-convoy cleanup for add-rate-limiting — MILESTONE, last P0 closed Reflects the merged add-rate-limiting convoy (PR #20, squash commit 708ef45) in repo documentation. **This is the milestone cleanup** — add-rate-limiting closed P0 #6 (No rate limiting anywhere), the LAST open P0 ship-blocker. `.convoys/ship-readiness.md`'s § Status summary flips from "7 of 8 RESOLVED; 1 remains" to **"8 of 8 RESOLVED. Launch-readiness P0 checklist is empty."** One brief in the convoy: Brief 1 shipped as planned with no scope expansions and no implementer deviations from the verbatim spec; all six architect decisions ratified verbatim at gate 1 (D1 operator-ratified Option A; D2-D6 architect-self-ratified). .convoys/add-rate-limiting.md: - frontmatter status: in-progress -> shipped (added shipped: 2026-05-24) - new ## As-shipped section. Opens with the milestone language pointing back at ship-readiness.md's flipped § Status summary. Decisions section captures all 6 ratifications (D1 operator- ratified Option A — Critical: WHY the atomic admin UI fix in pages/admin/card-import.js was Decision 1's hidden coupling requirement, since API gating alone would have broken every "Import Cards" click; D2 hybrid named-limiter shape with Map<className, Ratelimit> cache; D3 per-class table including the two D3 tuning-evidence raises — search 30 -> 60/min because ShareModal.handleSearch has no debounce so a 17-char email = 16 requests in <5s, and generate kept at 5/hour because DiceBear is free not paid AI; D4 two-extractor shape with defensive THROW on null/empty userId; D5 uniform 429 message; D6 no new vitest or playwright specs deferred to fill-vitest-handler-coverage). As-shipped surface broken into 4 layers (1 lib refactor + 6 route gates + 1 atomic admin UI fix + 1 rule extension) mirroring the cors-tighten cleanup's pattern-split shape. Empirical CI metrics from post-merge run 26382185019 (Playwright smoke 59s 3/3 in 3.8s, forbidden-cors-headers pass, vitest 21/21, lint 128 baseline, Screenshot diff continue-on-error swallow per Decision 4). Cross-validation finding: smoke test 2 still passes against the post-rate-limit preview — that's three convoys in a row (PR #15 Layout default-user, PR #19 CORS-tighten, PR #20 rate-limiting) where the same 3-test smoke spec defended the auth surface through sweeping changes. Operator-action-required: none. What did NOT change audit trail. .convoys/ship-readiness.md: - § Status summary at the top flipped from 7/8 to 8/8 RESOLVED. Header text updated to "Launch-readiness P0 checklist is empty." P0 #6 row in the table flips from PARTIAL to RESOLVED with the two-convoy lineage (fix-auth-bypass Brief 4 + add-rate-limiting). Trailing paragraph rewritten as a milestone note: security gate closed; remaining launch work is P1 quality bar + P2/P3 polish. - P0 #6 entry flipped from PARTIAL to RESOLVED with the full add-rate-limiting as-shipped block (8 sub-bullets covering the lib refactor shape, the per-class table, the defensive THROW, the three import routes' auth-gating, the atomic admin UI fix and WHY, the rule extension, the 6 decisions, and the diff breakdown). Brief 4's 2026-05-23 partial is preserved as the prior as-shipped layer to maintain the audit trail. - Launch sequence step 4 marked RESOLVED 2026-05-24 with the squash commit + smoke metrics inline. - Queued convoys: removed the add-rate-limiting entry (it shipped). Added a new delete-dead-lorcana-import entry (P3 polish; the Lorcana import route was gated defensively in PR #20 despite zero current frontend callers — pages/admin/card-import.js's <select> only offers mtg + pokemon — so if Lorcana stays permanently out of the admin UI, this is the cleanup PR). Added three "flagged but kept out of scope" follow-ups per the convoy's § What did NOT change: harden-multipart-parser (P2; 5MB body still consumed before the 429 path on avatar.js), god-function-split / refactor-cards-search-sql (P2; 240-line 7-branch SQL in cards/search.js), and withAdmin(handler) wrapper extraction (P3 DX; the three import routes are call sites #3-5 in the codebase but uniform inline shape was preserved for convoy atomicity). Updated tighten-visual-diff-path-filter to note PR #20 also tripped the same false-positive. AGENTS.md: - Gotcha #12 extended end-to-end. Was the single-class auth-only lib + the env-var contract; is now the 5-class reality with a full per-class table (helper / limit-window / key / routes), the defensive THROW pattern in extractUserIdentifier, the gate-ordering rule for per-user limiters, and the auth → admin-role → rate-limit ordering for the three import routes. Prominent milestone line opens the new content: "add-rate-limiting convoy (squash 708ef45, PR #20, 2026-05-24) closed P0 #6 — all 8 P0s now RESOLVED." Original env-var contract paragraph (KV_REST_API_URL / KV_REST_API_TOKEN, fail-closed-in-prod / warn-and-noop-in-dev) is preserved verbatim above the new content. - § 6 Testing: intentionally untouched (no test surface changed; vitest 21/21 and smoke 3/3 still apply). - § 7 Deployment: intentionally untouched (no deployment-shape changed; same KV_REST_API_* env vars from Brief 4). .cursor/rules/api-routes.mdc: - The implementer extended § Rate limiting in PR #20 with the per-class table + verbatim call shape + gate-ordering rules + identifier-extraction + uniform 429 + fail-closed env-var contract + fail-open Upstash-outage behavior. Doc-writer pass verified completeness; added a one-sentence convoy-attribution line at the top of § Rate limiting citing the two-convoy lineage (fix-auth-bypass Brief 4 for the auth class + add-rate-limiting for the other four classes and 7 newly-gated routes), mirroring the post-cors-tighten § CORS attribution shape. No other touch-ups needed. No changes to: package.json, package-lock.json, lib/rate-limit.js, pages/**, components/**, scripts/**, test/**, tests/**, .github/workflows/**, README.md, TESTING_GUIDE.md, playwright.config.js, eslint.config.mjs. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-25 00:09:07 -04:00
## As-shipped
Shipped 2026-05-24 as squash commit `708ef45` (PR #20, architect-commit
`60b842e`, implementer-commit `51a3a97`). **This is the milestone
convoy — it closes the LAST open P0 (P0 #6) and flips
`.convoys/ship-readiness.md`'s § Status summary from "7 of 8 RESOLVED;
1 remains" to "8 of 8 RESOLVED. Launch-readiness P0 checklist is
empty."** Brief 1 shipped as planned — single lib refactor + six
route-gate additions + one admin UI atomic fix + one rule extension,
no scope expansions beyond the architect-ratified Decision 1, and the
empirical CI metrics + the cross-validation finding + the
"D3 tuning-evidence" rationale captured here so the next architect /
reviewer has the audit trail.
### Decisions ratified at gate 1
All six decisions landed verbatim. Only D1 required operator gate-1
sign-off (significant scope expansion to admin-only enforcement on the
import surface plus an atomic admin UI touch); D2-D6 are
architect-self-ratifiable per the precedent established by
`cors-tighten` Decisions D2-D5 and
`fix-vercel-deployment-protection-in-ci` Decisions A/B/D.
- **Decision 1 — Option A (operator-ratified).** Gate all three
`pages/api/cards/import-*.js` routes in this convoy with
`getUserFromRequest` + `if (user.role !== 'admin') return 403` +
`checkImportRateLimit(req, user.userId)`. **And** atomically update
`pages/admin/card-import.js` to send
`'Authorization': \`Bearer ${localStorage.getItem('auth_token')}\``
on its import fetch — without this, the API gating would have
immediately broken every "Import Cards" click in the admin UI. The
architect's pre-brief investigation flagged this as the critical
hidden coupling: gating the import APIs without the matching client
fetch fix would have closed P0 #6 but introduced a visible UX
regression on the only live admin tooling that depends on it. Lorcana
was gated defensively despite having zero current frontend callers
(`pages/admin/card-import.js`'s `<select>` only offers `mtg` /
`pokemon`) — uniform protection across the three import shapes is
strictly easier to maintain than a 2-gated-1-deleted asymmetry, and
if Lorcana stays unused, the queued `delete-dead-lorcana-import`
follow-up convoy (see `.convoys/ship-readiness.md` § Queued
convoys) is the cleanup path.
- **Decision 2 — Hybrid named-limiter shape (architect-self-ratified).**
`lib/rate-limit.js` refactored from a single auth-only `Ratelimit`
instance into a `Map<className, Ratelimit>` cache with one shared
Redis client and five `Ratelimit` instances (one per class, distinct
Redis prefix). Five exported functions: `checkAuthRateLimit(req)`
(Brief 4 contract preserved byte-identical),
`checkSearchRateLimit(req)`, `checkUploadRateLimit(req, userId)`,
`checkGenerateRateLimit(req, userId)`, `checkImportRateLimit(req,
userId)`. Internal `check(className, identifier)` shared helper.
`LIMITER_CONFIG` is a top-level `const` map of
`{ limit, window, prefix }` per class; adding a sixth class is a
one-line addition + one new exported function (no `init()`
restructuring needed).
- **Decision 3 — Per-class limit values (architect-self-ratified with
tuning evidence).** Final table:
| Class | Limit | Window | Key | Helper |
| --- | --- | --- | --- | --- |
| `auth` | 5 | 15 min | IP | `checkAuthRateLimit(req)` (unchanged from Brief 4) |
| `search` | **60** | 1 min | IP | `checkSearchRateLimit(req)` |
| `upload` | 10 | 1 hour | user | `checkUploadRateLimit(req, userId)` |
| `generate` | **5** | 1 hour | user | `checkGenerateRateLimit(req, userId)` |
| `import` | 5 | 1 hour | user (admin-only) | `checkImportRateLimit(req, userId)` |
Two architect raises from the parent's pre-investigation defaults
warrant a permanent record so future tuning convoys see the
evidence:
- **`search` raised 30 → 60/min.** `components/ShareModal.js`'s
`handleSearch` (lines 56-77) fires on every keystroke with NO
debounce; typing a 17-char email = 16 requests in <5s, which
would 429 a single legitimate user against the 30/1min default.
60/1min covers a realistic burst and still stops a scraper
inside 2-3 seconds. If real users still 429, the fix is a
one-line `LIMITER_CONFIG` edit (60 → 90 or 120); not a
release-blocker. Queued follow-up name if needed:
`tune-search-rate-limit`.
- **`generate` kept at 5/hour (parent suggested 3).**
`pages/api/user/avatar/generate.js` calls DiceBear (free public
API), not OpenAI / Replicate / Stability — cost is Vercel blob
storage + DiceBear-side throttling, not per-call $. 5/hour is
generous enough that a user trying 3-4 seeds doesn't hit the
wall, strict enough that an accidental render loop still trips
inside the first minute. If we ever switch generators to a paid
provider, the same one-line `LIMITER_CONFIG` edit drops it back
to 3 or lower.
- **Decision 4 — Two-extractor shape with defensive THROW
(architect-self-ratified).** `extractIpIdentifier(req)`
(module-private; renamed from the pre-refactor `extractIdentifier`)
and `extractUserIdentifier(userId)` (module-private; new). The
user extractor **THROWS** with
`'[rate-limit] extractUserIdentifier called without an
authenticated userId. Place the rate-limit gate AFTER the auth
check, never before.'` when `userId` is `null` / `undefined` /
`''` / `NaN`. Surfaces gate-ordering bugs **at dev time** rather
than silently falling back to IP and converting a per-user limit
into a per-IP limit (which would lock other household members out
for one user's behavior — the exact regression flagged in the
convoy file's § Known constraints). Numeric `0` is intentionally
accepted (returns `'user:0'`) — defensively forward-compatible if
a future schema introduces user ID 0.
- **Decision 5 — Uniform 429 response (architect-self-ratified).**
Single `'Too many attempts. Try again later.'` message across all
five classes, matching `login.js` + `register.js` verbatim.
`Retry-After` calculation is `Math.ceil((reset - Date.now()) /
1000)`; status code is `429`. Per-class variation (e.g., *"Too
many search requests"*) was considered and rejected because it
would fingerprint to an attacker which routes have which limits +
windows, making it easier to craft a request pattern that avoids
429 on the more-permissive routes while still abusing the
less-permissive ones.
- **Decision 6 — No new vitest or playwright tests
(architect-self-ratified).** Same reasoning as `cors-tighten`
Decision D4. Per-route handler tests are deferred to the queued
`fill-vitest-handler-coverage` convoy. The fail-loud-in-prod
predicate in `lib/rate-limit.js::init()` is the integration test
— if `KV_REST_API_*` is unset on a deployed env, every gated route
fails closed on the first call. Architect-time investigation
corrected a stale claim in the convoy file's § Known constraints
(the line *"auth-utils tests run through the same module"*): `rg
'rate-limit|@upstash' test/` returns zero matches, so the lib
refactor is strictly safer than the convoy file implied — there
is no transitive vitest path to break.
### As-shipped surface
The shape splits cleanly into four layers (mirrors `cors-tighten`'s
24-source-files / 1-CI-job pattern split, but with one extra layer
because of the atomic admin UI fix and the rule extension):
1. **One `lib/rate-limit.js` refactor.** Single-class auth-only
limiter → 5-class `Map<className, Ratelimit>` with distinct Redis
prefixes (`tcgvault:auth`, `tcgvault:search`, `tcgvault:upload`,
`tcgvault:generate`, `tcgvault:import`). +90 / -23 net (90 lines
added, 23 lines reshaped — the existing `init()`,
`extractIdentifier`, `checkAuthRateLimit` functions are
conceptually preserved but restructured to share infrastructure
across all five classes). Brief 4's `checkAuthRateLimit(req)`
return shape is **byte-identical** post-refactor (`{ allowed,
remaining, reset }`); `login.js` + `register.js` were not
touched and continue to work unchanged.
2. **Six new route-gate additions** (lib refactor delivers the
helpers; routes call them):
- `pages/api/users/search.js``checkSearchRateLimit(req)` after
the existing inline JWT verify. IP-keyed. +7 lines.
- `pages/api/cards/search.js``checkSearchRateLimit(req)` at top
of handler after the method check, before the 240-line SQL
god-function (which stays byte-identical per § Files explicitly
out of scope). IP-keyed, anonymous-by-design. +7 lines.
- `pages/api/user/avatar.js` — `checkUploadRateLimit(req,
user.userId)` after `getUserFromRequest`, before the POST/DELETE
method branching (so the limiter fires before
`parseMultipartFormData` consumes the 5MB body). User-keyed. +7
lines.
- `pages/api/user/avatar/generate.js`
`checkGenerateRateLimit(req, user.userId)` after
`getUserFromRequest`, before the user-data SQL query.
User-keyed. +7 lines.
- `pages/api/cards/import-mtg.js`
`getUserFromRequest` + `if (user.role !== 'admin') return 403`
+ `checkImportRateLimit(req, user.userId)` before the existing
`try` block. Closes the publicly-callable anonymous-abuse
vector. +16 lines.
- `pages/api/cards/import-pokemon.js` — same shape as
`import-mtg.js`. +16 lines.
- `pages/api/cards/import-lorcana.js` — same shape as
`import-mtg.js`; gated defensively despite zero current
frontend callers. +16 lines.
3. **One atomic admin UI fix.** `pages/admin/card-import.js` adds
`'Authorization': \`Bearer ${localStorage.getItem('auth_token')}\``
to the import fetch's headers (one-line addition; the 309-line
god-component is otherwise byte-identical). **This was the
architect's critical pre-brief discovery and the reason Decision
1 routes back to the operator** — gating the import APIs without
this matching client fetch fix would have closed P0 #6 but
introduced an immediate 401 on every "Import Cards" click,
producing a visible UX regression on the only live admin tooling
that exercises the gated routes. Shipping the gate + the client
fix in the same PR keeps the convoy atomic; PR review enforced
the dependency explicitly.
4. **One `.cursor/rules/api-routes.mdc` § Rate limiting extension.**
Replaced the previous auth-only § with the per-class table + the
verbatim call shape + gate-ordering rules (method check first;
auth before any user-keyed limiter; IP-keyed gate placement is
flexible; admin-role check goes between auth and rate-limit for
the import routes) + identifier-extraction documentation +
uniform 429 response shape + fail-closed env-var contract +
fail-open Upstash-outage behavior. Implementer landed this in
the same PR; doc-writer pass verified the extension is complete
and made no further touch-ups (see § What did NOT change below).
### As-shipped metrics
Diff size (per `git show --stat 708ef45`):
- **12 files modified, +1612 / -23.** Note that the 1612-addition
figure is dominated by `.convoys/add-rate-limiting.md` (676 lines)
and `.convoys/add-rate-limiting/brief-1-extend-rate-limit-and-wire-routes.md`
(751 lines), which the squash includes because the architect commit
preceded the implementer commit on the same branch. The actual
source-file diff is much smaller:
- `lib/rate-limit.js`: +90 / -23 (the lib refactor).
- `.cursor/rules/api-routes.mdc`: +41 (the § Rate limiting extension).
- 6 route files under `pages/api/`: +69 lines total (3× +16 for
import routes, 4× +7 for search/avatar/generate/users-search).
- `pages/admin/card-import.js`: +1 (the Bearer-header addition).
Post-merge CI run 26382185019 + subsequent runs on `main`:
- **`Playwright smoke` — PASS in 59s, 3/3 tests in 3.8s** against the
post-rate-limit Vercel preview. Same three checks (`home redirects
or renders without 5xx` ✓ 431ms / `sign-in page renders` ✓ 331ms /
`public health endpoint responds` ✓ 193ms) — all green. **Critical
cross-validation:** smoke calls `/api/health` once per run (well
below the search limiter's 60/min ceiling), and the home + sign-in
routes don't touch any of the 6 newly-gated endpoints, so smoke
does NOT 429 against the new search class. The cross-validation
finding accumulates: smoke test 2 (`'sign-in page renders'`) still
passes against the post-CORS + post-rate-limit preview — that's
three convoys in a row (PR #15 Layout default-user → PR #19
CORS-tighten → PR #20 rate-limiting) where the auth surface stayed
stable under sweeping changes, and the same 3-test smoke spec
defended it every time.
- **`forbidden-cors-headers` (from `cors-tighten`)** — PASS. None of
the 6 route edits introduced an `Access-Control-Allow-*` header
(the convoy is purely additive of rate-limit gate code; CORS
surface was not touched). The grep stays clean.
- **`forbidden-endpoints` (from `fix-auth-bypass` Brief 3)** — PASS.
No new `pages/api/test-*.js` or other deleted-endpoint shapes
reintroduced.
- **`Unit tests (vitest)`** — PASS, 21/21 in 27s. No new tests, no
removed tests; the existing `lib/auth-secret.test.js` (3) +
`lib/permission-middleware.test.js` (8) +
`pages/api/auth-utils.test.js` (5) +
`components/Layout.test.js` (5) suites are all unaffected. Decision
6's architect-time verification (`rg 'rate-limit|@upstash' test/`
returns zero matches) confirmed at merge.
- **`Lint`** — 128 problems (lint baseline preserved, no
regression). The lib refactor + the 6 route edits + the admin UI
one-liner introduced zero new lint problems; the `|| true`
wrapper in `.github/workflows/ci.yml` was a no-op for this convoy.
- **`Screenshot diff`** — `continue-on-error: true` swallow per
`adopt-playwright-smoke` Decision 4 (no baseline committed yet);
the documented Decision-4 end state. Triggered on PR #20 because
the `paths:` filter `pages/**` matches the 6 route edits under
`pages/api/`; same minor false-positive as PR #19, tracked by the
queued `tighten-visual-diff-path-filter` follow-up.
- **All other gates (`Schema map up to date`, `Aggregate gate`)** —
green.
### Cross-validation finding: smoke spec defends the rate-limit
surface (organically)
The 3-test smoke spec was authored by `adopt-playwright-smoke` (PR
#18) against an un-rate-limited preview, with no foresight about
this convoy's gating. Post-merge run 26382185019 confirms the spec
**still passes against the post-rate-limit preview** — the home
route, the `/login` route, and `/api/health` don't touch any of the
six newly-gated endpoints, and `/api/health` is anonymous /
unrate-limited so the smoke flow doesn't bump up against the search
class's 60/min ceiling. The cross-validation lineage accumulates:
- PR #15 (`fix-layout-default-user`, `ca302a8`) introduced the
`<Link href="/login">Sign in</Link>` CTA that smoke test 2 asserts
on.
- PR #19 (`cors-tighten`, `da50d78`) removed wildcard CORS from 24
handlers; smoke test 2 still passed against the post-CORS
preview.
- PR #20 (`add-rate-limiting`, `708ef45`) — this convoy — wired 6 new
route gates; smoke test 2 still passes.
Smoke is doing real work: it has now defended the auth surface
against three sweeping changes without anyone having to write a
single dedicated test. P0 #7's resolved state, P0 #5's resolved
state, and now P0 #6's resolved state are all backed by a live CI
signal — not just a vitest assertion.
### Operator action required going forward
**None.** No new env vars (Upstash `KV_REST_API_URL` /
`KV_REST_API_TOKEN` were already auto-provisioned via the Vercel
Marketplace integration for Brief 4). No new dependencies
(`@upstash/ratelimit@^2.0.8` + `@upstash/redis@^1.38.0` already
installed). No new secrets, no infra changes, no CI gates to enable
or disable. The fail-loud-in-prod predicate in
`lib/rate-limit.js::init()` is self-defending: if a future deploy
unsets either env var, every gated route fails closed on the first
call (`throw new Error('[rate-limit] Upstash not configured...')`),
which surfaces immediately as a 500 in the Vercel logs rather than
silently disabling brute-force protection.
If a follow-up tuning need surfaces (search 60/min too tight, generate
5/hour too tight, etc.), the fix is a single-line `LIMITER_CONFIG`
edit. See `tune-search-rate-limit` (not yet queued; surface only if
real users 429) and the broader `tiered-rate-limits` (P3 polish,
when a premium-tier scheme exists) for the longer view.
### What did NOT change
Audit trail of files explicitly NOT touched by this convoy, despite
sitting near the rate-limit surface:
- **`pages/api/auth/login.js`, `pages/api/auth/register.js`** —
Brief 4 contract preserved. Both files continue to call
`checkAuthRateLimit(req)` against the refactored lib; byte-identical
post-merge. Manual verification at architect time: the 6th-attempt
429 path still fires correctly.
- **`lib/permission-middleware.js`, `lib/auth-secret.js`,
`pages/api/auth-utils.js`** — auth surface untouched. No `null`-vs-
synthetic-admin regression risk; `test/lib/permission-middleware.test.js`'s
negative regression test still defends Gotcha #2.
- **`package.json`, `package-lock.json`** — zero dep additions, zero
version bumps. `@upstash/ratelimit@^2.0.8` + `@upstash/redis@^1.38.0`
pinned from Brief 4.
- **`.github/workflows/*.yml`** — no new CI gate. Per-class
rate-limit wiring isn't grep-checkable; the fail-loud-in-prod
predicate is the integration test (see Decision 6).
- **`test/**`** — Decision 6, no new vitest or playwright specs.
21/21 still green; smoke 3/3 still green.
- **`pages/api/cards/search.js`'s SQL** — the 240-line god-function
with 7+ conditional `SELECT * FROM cards WHERE …` branches was
explicitly out of scope per `.convoys/add-rate-limiting.md` §
Out of scope. Queued as `god-function-split` /
`refactor-cards-search-sql` (not yet queued in `ship-readiness.md`
— surface when the convoy is sized).
- **`pages/api/user/avatar.js`'s `parseMultipartFormData` body
streaming** — the 5MB multipart body is consumed via
`req.on('data')` before any rate-limit gate can short-circuit, so
an attacker can still exhaust the 5MB body per 429. Queued as
`harden-multipart-parser` (not yet in `ship-readiness.md` — surface
if a real abuse incident occurs). The brief's gate-ordering places
the limiter BEFORE the method branches that call
`parseMultipartFormData`, so when that future hardening lands, the
gate ordering is already correct.
- **`scripts/import-*.js`** — standalone CLI scripts independent of
the API routes; not in scope per § Files explicitly out of scope.
An operator running `node scripts/import-mtg.js` directly bypasses
the rate-limit + admin-role gate entirely (which is the intended
flow — local admin work is unthrottled).
- **`AGENTS.md`'s Gotcha #5 (deleted `setup-database.js`) + Gotcha
#8 (Layout default-user)** — both still RESOLVED, both unchanged.
This convoy's doc-writer pass updates Gotcha #12 (rate-limit
env-var contract) to reflect the 5-class reality, but Gotcha #5
and #8 are independent of this convoy.
- **Auth wrapper extraction (`withAdmin(handler)`)** — flagged in
`.cursor/rules/auth-and-permissions.mdc` (*"check user.role ===
'admin' directly; consider extracting `withAdmin()` if a third call
site appears"*). The three import routes are the third+fourth+fifth
call sites in the codebase, but extracting the wrapper is its own
scope — for this convoy, the inline `if (user.role !== 'admin')
return 403` shape was preserved across all three. Surface as a
follow-up convoy if a sixth call site appears or if a reviewer
flags the inline shape as a maintainability concern; for now, the
uniform inline check across the three import routes is consistent
with the rest of the codebase.