rstillwell/deckhearth
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Remove dead Lorcana import route and CLI script #59

Merged
varutasu merged 2 commits from chore/delete-dead-lorcana-import into main 2026-06-02 01:42:18 -04:00
Conversation 2 Commits 2 Files changed 6 +4 -451
6 changed files with 4 additions and 451 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
.cursor/rules/api-routes.mdc
View file

@ -111,7 +111,7 @@ await logCollectionActivity(collectionId, userId, 'card_added', { cardId, quanti
| `search` | 60 | 1 min | IP | `/api/users/search`, `/api/cards/search` | `checkSearchRateLimit(req)` | | `search` | 60 | 1 min | IP | `/api/users/search`, `/api/cards/search` | `checkSearchRateLimit(req)` |
| `upload` | 10 | 1 hour | user | `/api/user/avatar` | `checkUploadRateLimit(req, userId)` | | `upload` | 10 | 1 hour | user | `/api/user/avatar` | `checkUploadRateLimit(req, userId)` |
| `generate` | 5 | 1 hour | user | `/api/user/avatar/generate` | `checkGenerateRateLimit(req, userId)` | | `generate` | 5 | 1 hour | user | `/api/user/avatar/generate` | `checkGenerateRateLimit(req, userId)` |
| `import` | 5 | 1 hour | user | `/api/cards/import-mtg`, `/api/cards/import-pokemon`, `/api/cards/import-lorcana` | `checkImportRateLimit(req, userId)` | | `import` | 5 | 1 hour | user | `/api/cards/import-mtg`, `/api/cards/import-pokemon` | `checkImportRateLimit(req, userId)` |
| `scan` | 5 | 1 min | user | `/api/scan/identify` | `checkScanRateLimit(req, userId)` | | `scan` | 5 | 1 min | user | `/api/scan/identify` | `checkScanRateLimit(req, userId)` |
**Verbatim call shape** (identical across all six classes — only the helper name and the optional `userId` argument differ): **Verbatim call shape** (identical across all six classes — only the helper name and the optional `userId` argument differ):
@ -145,7 +145,7 @@ export default async function handler(req, res) {
1. **Method check first.** Reject the wrong verb with 405 before doing any limiter work. 1. **Method check first.** Reject the wrong verb with 405 before doing any limiter work.
2. **Auth check before any user-keyed limiter.** `extractUserIdentifier(userId)` THROWS when `userId` is null/undefined/empty (defensive). For `upload`, `generate`, `import`, and `scan`, the handler MUST call `getUserFromRequest(req)` (or equivalent JWT verification) and confirm a non-null user BEFORE calling the limiter. Wrong order = anonymous user bypasses (the THROW surfaces immediately during dev; do not catch and silently fall back to IP). 2. **Auth check before any user-keyed limiter.** `extractUserIdentifier(userId)` THROWS when `userId` is null/undefined/empty (defensive). For `upload`, `generate`, `import`, and `scan`, the handler MUST call `getUserFromRequest(req)` (or equivalent JWT verification) and confirm a non-null user BEFORE calling the limiter. Wrong order = anonymous user bypasses (the THROW surfaces immediately during dev; do not catch and silently fall back to IP).
3. **For IP-keyed limiters (`auth`, `search`), gate placement is flexible** — either at the top of the handler (after the method check) or after a separate auth check that the route happens to also have (e.g. `users/search` JWT-verifies before rate-limiting, both are correct). The limiter only needs `req` for IP extraction. 3. **For IP-keyed limiters (`auth`, `search`), gate placement is flexible** — either at the top of the handler (after the method check) or after a separate auth check that the route happens to also have (e.g. `users/search` JWT-verifies before rate-limiting, both are correct). The limiter only needs `req` for IP extraction.
4. **Admin-role check, if applicable, goes between auth and rate-limit.** Used by all three `/api/cards/import-*` routes: `if (user.role !== 'admin') return res.status(403).json({ error: 'Admin access required' })` sits between the `if (!user)` 401 and the import rate-limit call. 4. **Admin-role check, if applicable, goes between auth and rate-limit.** Used by both `/api/cards/import-*` routes: `if (user.role !== 'admin') return res.status(403).json({ error: 'Admin access required' })` sits between the `if (!user)` 401 and the import rate-limit call.
**Identifier extraction:** **Identifier extraction:**

4
AGENTS.md
View file

@ -101,13 +101,13 @@ Code graph is indexed by `user-code-review-graph` MCP (122 files, 628 nodes, 560
| `checkSearchRateLimit(req)` | `search` | 60 / 1 min | IP | `deckhearth:search` | `/api/users/search`, `/api/cards/search` | | `checkSearchRateLimit(req)` | `search` | 60 / 1 min | IP | `deckhearth:search` | `/api/users/search`, `/api/cards/search` |
| `checkUploadRateLimit(req, userId)` | `upload` | 10 / 1 hour | user | `deckhearth:upload` | `/api/user/avatar` | | `checkUploadRateLimit(req, userId)` | `upload` | 10 / 1 hour | user | `deckhearth:upload` | `/api/user/avatar` |
| `checkGenerateRateLimit(req, userId)` | `generate` | 5 / 1 hour | user | `deckhearth:generate` | `/api/user/avatar/generate` | | `checkGenerateRateLimit(req, userId)` | `generate` | 5 / 1 hour | user | `deckhearth:generate` | `/api/user/avatar/generate` |
| `checkImportRateLimit(req, userId)` | `import` | 5 / 1 hour | user (admin-only) | `deckhearth:import` | `/api/cards/import-mtg`, `/api/cards/import-pokemon`, `/api/cards/import-lorcana` | | `checkImportRateLimit(req, userId)` | `import` | 5 / 1 hour | user (admin-only) | `deckhearth:import` | `/api/cards/import-mtg`, `/api/cards/import-pokemon` |
Prefixes renamed `tcgvault:*` → `deckhearth:*` in `pick-a-name` (squash `9abbab6`, 2026-05-24); accepted one-time per-15-min / per-1-hour counter reset; existing Upstash state at `tcgvault:*` keys is now stale and will TTL out naturally. Prefixes renamed `tcgvault:*` → `deckhearth:*` in `pick-a-name` (squash `9abbab6`, 2026-05-24); accepted one-time per-15-min / per-1-hour counter reset; existing Upstash state at `tcgvault:*` keys is now stale and will TTL out naturally.
All five return the same `{ allowed, remaining, reset }` shape; on `!allowed`, set `Retry-After: Math.ceil((reset - Date.now()) / 1000)` and return 429 with the uniform message `'Too many attempts. Try again later.'` (per-class variation would fingerprint the limits to an attacker — explicitly rejected). All five return the same `{ allowed, remaining, reset }` shape; on `!allowed`, set `Retry-After: Math.ceil((reset - Date.now()) / 1000)` and return 429 with the uniform message `'Too many attempts. Try again later.'` (per-class variation would fingerprint the limits to an attacker — explicitly rejected).
**Defensive THROW pattern.** `extractUserIdentifier(userId)` THROWS with a named error when `userId` is `null` / `undefined` / `''` / `NaN`. Surfaces gate-ordering bugs at dev time rather than silently falling back to IP and converting a per-user limit into a per-IP limit (which would lock household members out for one user's behavior). Numeric `0` is intentionally accepted (returns `'user:0'`) for forward-compat. **Gate-ordering rule: per-user rate-limit gates (`upload`, `generate`, `import`) MUST sit AFTER the auth check.** For the three `/api/cards/import-*` routes, the ordering is also `auth → admin-role check (403 if not admin) → rate-limit`; the admin-role check sits between auth and rate-limit. IP-keyed gates (`auth`, `search`) can sit anywhere after the method check. **Defensive THROW pattern.** `extractUserIdentifier(userId)` THROWS with a named error when `userId` is `null` / `undefined` / `''` / `NaN`. Surfaces gate-ordering bugs at dev time rather than silently falling back to IP and converting a per-user limit into a per-IP limit (which would lock household members out for one user's behavior). Numeric `0` is intentionally accepted (returns `'user:0'`) for forward-compat. **Gate-ordering rule: per-user rate-limit gates (`upload`, `generate`, `import`) MUST sit AFTER the auth check.** For the two `/api/cards/import-*` routes, the ordering is also `auth → admin-role check (403 if not admin) → rate-limit`; the admin-role check sits between auth and rate-limit. IP-keyed gates (`auth`, `search`) can sit anywhere after the method check.
Adding a sixth class is a one-line `LIMITER_CONFIG` addition + one new exported function (no `init()` restructuring needed). Tuning an existing class is a one-line `LIMITER_CONFIG` edit. The full verbatim call shape + gate-ordering rules + identifier-extraction documentation live in `.cursor/rules/api-routes.mdc` § Rate limiting. Adding a sixth class is a one-line `LIMITER_CONFIG` addition + one new exported function (no `init()` restructuring needed). Tuning an existing class is a one-line `LIMITER_CONFIG` edit. The full verbatim call shape + gate-ordering rules + identifier-extraction documentation live in `.cursor/rules/api-routes.mdc` § Rate limiting.

185
pages/api/cards/import-lorcana.js
View file

@ -1,185 +0,0 @@
import { sql } from '@vercel/postgres';
import { getUserFromRequest } from '../../../lib/permission-middleware';
import { checkImportRateLimit } from '../../../lib/rate-limit.js';
// Helper function to delay execution
const delay = (ms) => new Promise(resolve => setTimeout(resolve, ms));
// Helper function to fetch with retry logic
async function fetchWithRetry(url, maxRetries = 3, delayMs = 2000) {
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const response = await fetch(url, {
headers: {
'User-Agent': 'Deck-Hearth/1.0',
'Accept': 'application/json'
},
timeout: 30000 // 30 second timeout
});
if (response.ok) {
return response;
}
// If it's a 404, don't retry
if (response.status === 404) {
throw new Error(`Set not found: ${response.status}`);
}
// If it's a 504 or 503, wait longer before retry
if (response.status === 504 || response.status === 503) {
console.log(`Attempt ${attempt}: Got ${response.status}, waiting ${delayMs * attempt}ms before retry...`);
await delay(delayMs * attempt);
continue;
}
throw new Error(`Lorcana API error: ${response.status}`);
} catch (error) {
if (attempt === maxRetries) {
throw error;
}
console.log(`Attempt ${attempt} failed:`, error.message);
await delay(delayMs * attempt);
}
}
}
export default async function handler(req, res) {
if (req.method !== 'POST') {
return res.status(405).json({ error: 'Method not allowed' });
}
const user = await getUserFromRequest(req);
if (!user) {
return res.status(401).json({ error: 'Authentication required' });
}
if (user.role !== 'admin') {
return res.status(403).json({ error: 'Admin access required' });
}
const { allowed, reset } = await checkImportRateLimit(req, user.userId);
if (!allowed) {
res.setHeader('Retry-After', Math.ceil((reset - Date.now()) / 1000));
return res.status(429).json({ error: 'Too many attempts. Try again later.' });
}
try {
const { setCode } = req.body;
if (!setCode) {
return res.status(400).json({ error: 'Set code is required' });
}
console.log(`Starting import for Lorcana set: ${setCode}`);
// Map set codes to Lorcast set codes
const setCodeMap = {
'tfc': '1', // The First Chapter
'rotf': '2', // Rise of the Floodborn
'ink': '3' // Into the Inklands
};
const lorcastSetCode = setCodeMap[setCode];
if (!lorcastSetCode) {
return res.status(400).json({ error: `Unknown set code: ${setCode}` });
}
// Fetch cards from Lorcast API
const response = await fetchWithRetry(`https://api.lorcast.com/v0/cards/search?q=set:${lorcastSetCode}&unique=prints`);
const data = await response.json();
const cards = data.results || [];
if (cards.length === 0) {
return res.status(200).json({
success: true,
message: `No cards found for set ${setCode}`,
imported: 0,
skipped: 0,
total: 0
});
}
console.log(`Found ${cards.length} cards for set ${setCode}`);
let importedCount = 0;
let skippedCount = 0;
for (const card of cards) {
try {
// Check if card already exists
const existingCard = await sql`
SELECT id FROM cards WHERE scryfall_id = ${card.id}
`;
if (existingCard.rows.length > 0) {
skippedCount++;
continue;
}
// Extract price data
let currentPrice = null;
if (card.prices?.usd) {
currentPrice = parseFloat(card.prices.usd);
} else if (card.prices?.usd_foil) {
currentPrice = parseFloat(card.prices.usd_foil);
}
// Determine rarity
let rarity = card.rarity || 'Common';
if (rarity === 'Super_rare') {
rarity = 'Super Rare';
}
// Extract card type and text
const cardType = card.type?.[0] || 'Character';
const cardText = card.text || null;
const flavorText = card.flavor_text || null;
// Extract stats
const strength = card.strength || null;
const willpower = card.willpower || null;
const lore = card.lore || null;
// Insert card into database
await sql`
INSERT INTO cards (
name, set_name, set_code, card_number, rarity, game,
mana_cost, cmc, card_type, colors, oracle_text,
power, toughness, image_url, stock_image_url,
current_price, market_price, scryfall_id, verified
) VALUES (
${card.name}, ${card.set.name}, ${card.set.code}, ${card.collector_number},
${rarity}, 'Lorcana', ${card.cost}, ${card.cost}, ${cardType},
${JSON.stringify(card.ink ? [card.ink] : [])}, ${cardText || flavorText},
${strength}, ${willpower || lore},
${card.image_uris?.digital?.large || card.image_uris?.digital?.small || null}, ${card.image_uris?.digital?.small || null},
${currentPrice}, null, ${card.id}, true
)
`;
importedCount++;
} catch (error) {
console.error(`Error importing card ${card.name}:`, error);
skippedCount++;
}
}
console.log(`Import completed for set ${setCode}: ${importedCount} imported, ${skippedCount} skipped`);
res.status(200).json({
success: true,
message: `Import completed for set ${setCode}`,
imported: importedCount,
skipped: skippedCount,
total: cards.length
});
} catch (error) {
console.error('Card import error:', error);
res.status(500).json({
error: 'Import failed',
details: error.message
});
}
}

4
scripts/README.md
View file

@ -17,7 +17,6 @@ npm run import-popular
**What it imports:** **What it imports:**
- **Magic: The Gathering**: ~100+ sets from Alpha to recent releases - **Magic: The Gathering**: ~100+ sets from Alpha to recent releases
- **Pokemon**: ~100+ sets from Base Set to current Scarlet & Violet - **Pokemon**: ~100+ sets from Base Set to current Scarlet & Violet
- **Lorcana**: All 3 available sets
**Estimated time:** 2-4 hours (depending on API response times) **Estimated time:** 2-4 hours (depending on API response times)
@ -32,7 +31,6 @@ npm run import-all
**What it imports:** **What it imports:**
- **Magic: The Gathering**: 100+ sets (Alpha to current) - **Magic: The Gathering**: 100+ sets (Alpha to current)
- **Pokemon**: 100+ sets (Base Set to current) - **Pokemon**: 100+ sets (Base Set to current)
- **Lorcana**: All available sets
**Estimated time:** 4-8 hours (depending on API response times) **Estimated time:** 4-8 hours (depending on API response times)
@ -171,13 +169,11 @@ You can stop the script with `Ctrl+C` and restart it later. The scripts will sta
- `POST /api/cards/import-mtg` - Magic: The Gathering imports - `POST /api/cards/import-mtg` - Magic: The Gathering imports
- `POST /api/cards/import-pokemon` - Pokemon imports - `POST /api/cards/import-pokemon` - Pokemon imports
- `POST /api/cards/import-lorcana` - Lorcana imports
## Data Sources ## Data Sources
- **Magic: The Gathering**: Scryfall API - **Magic: The Gathering**: Scryfall API
- **Pokemon**: [PokemonTCG/pokemon-tcg-data](https://github.com/PokemonTCG/pokemon-tcg-data) on GitHub (sets + per-set JSON; images from linked CDNs). Optional override: `POKEMON_TCG_DATA_BASE_URL` (defaults to `master` branch raw URLs). The legacy Pokemon TCG API is no longer used by catalog sync. - **Pokemon**: [PokemonTCG/pokemon-tcg-data](https://github.com/PokemonTCG/pokemon-tcg-data) on GitHub (sets + per-set JSON; images from linked CDNs). Optional override: `POKEMON_TCG_DATA_BASE_URL` (defaults to `master` branch raw URLs). The legacy Pokemon TCG API is no longer used by catalog sync.
- **Lorcana**: Lorcana API (limited availability)
## Performance Notes ## Performance Notes

98
scripts/import-lorcana-simple.js
View file

@ -1,98 +0,0 @@
import fetch from 'node-fetch';
import fs from 'fs';
// Lorcana sets to import
const LORCANA_SETS = [
{ code: 'tfc', name: 'The First Chapter' },
{ code: 'rotf', name: 'Rise of the Floodborn' },
{ code: 'ink', name: 'Into the Inklands' }
];
async function importLorcanaSet(setCode, setName) {
try {
console.log(`Importing Lorcana set: ${setName} (${setCode})`);
const response = await fetch(`http://localhost:3000/api/cards/import-lorcana`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
setCode: setCode,
setName: setName
})
});
const result = await response.json();
if (result.success) {
console.log(`✅ Successfully imported ${result.imported || 0} cards from ${setName}`);
return { success: true, importedCount: result.imported || 0 };
} else {
console.log(`❌ Failed to import ${setName}: ${result.error}`);
return { success: false, error: result.error };
}
} catch (error) {
console.log(`❌ Error importing ${setName}: ${error.message}`);
return { success: false, error: error.message };
}
}
async function importAllLorcana() {
console.log('🎨 Starting Lorcana card import...\n');
const results = {
total: 0,
successful: 0,
failed: 0,
sets: []
};
for (const set of LORCANA_SETS) {
const result = await importLorcanaSet(set.code, set.name);
results.sets.push({ ...set, ...result });
if (result.success) {
results.successful++;
results.total += result.importedCount || 0;
} else {
results.failed++;
}
// Add a delay between sets
await new Promise(resolve => setTimeout(resolve, 2000));
}
// Print summary
console.log('\n📊 Lorcana Import Summary:');
console.log('========================');
console.log(`Total sets: ${LORCANA_SETS.length}`);
console.log(`Successful: ${results.successful}`);
console.log(`Failed: ${results.failed}`);
console.log(`Total cards imported: ${results.total}`);
console.log(`Success rate: ${((results.successful / LORCANA_SETS.length) * 100).toFixed(1)}%`);
// Save detailed results
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const filename = `lorcana-import-results-${timestamp}.json`;
fs.writeFileSync(filename, JSON.stringify(results, null, 2));
console.log(`\n📄 Detailed results saved to: ${filename}`);
return results;
}
// Run the import if this script is executed directly
if (import.meta.url === `file://${process.argv[1]}`) {
importAllLorcana()
.then(() => {
console.log('\n🎉 Lorcana import completed!');
process.exit(0);
})
.catch((error) => {
console.error('❌ Import failed:', error);
process.exit(1);
});
}
export { importAllLorcana, importLorcanaSet };

160
scripts/import-lorcana.js
View file

@ -1,160 +0,0 @@
import fetch from 'node-fetch';
import fs from 'fs';
// Lorcana sets to import
const LORCANA_SETS = [
{ code: 'tfc', name: 'The First Chapter', lorcastCode: '1' },
{ code: 'rotf', name: 'Rise of the Floodborn', lorcastCode: '2' },
{ code: 'ink', name: 'Into the Inklands', lorcastCode: '3' }
];
// Helper function to delay execution
const delay = (ms) => new Promise(resolve => setTimeout(resolve, ms));
// Helper function to fetch with retry logic
async function fetchWithRetry(url, maxRetries = 3, delayMs = 2000) {
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const response = await fetch(url, {
headers: {
'User-Agent': 'Deck-Hearth/1.0',
'Accept': 'application/json'
},
timeout: 30000 // 30 second timeout
});
if (response.ok) {
return response;
}
// If it's a 404, don't retry
if (response.status === 404) {
throw new Error(`Set not found: ${response.status}`);
}
// If it's a 504 or 503, wait longer before retry
if (response.status === 504 || response.status === 503) {
console.log(`Attempt ${attempt}: Got ${response.status}, waiting ${delayMs * attempt}ms before retry...`);
await delay(delayMs * attempt);
continue;
}
throw new Error(`Lorcana API error: ${response.status}`);
} catch (error) {
if (attempt === maxRetries) {
throw error;
}
console.log(`Attempt ${attempt} failed:`, error.message);
await delay(delayMs * attempt);
}
}
}
async function importLorcanaSet(setCode, setName, lorcastCode) {
try {
console.log(`\n🎨 Importing Lorcana set: ${setName} (${setCode})`);
// Fetch cards directly from Lorcast API
const response = await fetchWithRetry(`https://api.lorcast.com/v0/cards/search?q=set:${lorcastCode}&unique=prints`);
const data = await response.json();
const cards = data.results || [];
if (cards.length === 0) {
console.log(`❌ No cards found for set ${setName}`);
return { success: false, error: 'No cards found', importedCount: 0 };
}
console.log(`📊 Found ${cards.length} cards for set ${setName}`);
let importedCount = 0;
let skippedCount = 0;
// Import all cards via our API in a single batch
const importResponse = await fetch('http://localhost:3000/api/cards/import-lorcana', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
setCode: setCode,
setName: setName
})
});
const importResult = await importResponse.json();
if (importResult.success) {
importedCount = importResult.imported || 0;
skippedCount = importResult.skipped || 0;
console.log(`✅ Import completed for ${setName}: ${importedCount} imported, ${skippedCount} skipped`);
} else {
console.log(`❌ Failed to import ${setName}: ${importResult.error}`);
return { success: false, error: importResult.error };
}
return { success: true, importedCount, skippedCount, total: cards.length };
} catch (error) {
console.log(`❌ Error importing ${setName}: ${error.message}`);
return { success: false, error: error.message };
}
}
async function importAllLorcana() {
console.log('🎨 Starting Lorcana card import...\n');
const results = {
total: 0,
successful: 0,
failed: 0,
sets: []
};
for (const set of LORCANA_SETS) {
const result = await importLorcanaSet(set.code, set.name, set.lorcastCode);
results.sets.push({ ...set, ...result });
if (result.success) {
results.successful++;
results.total += result.importedCount || 0;
} else {
results.failed++;
}
// Add a delay between sets
await delay(2000);
}
// Print summary
console.log('\n📊 Lorcana Import Summary:');
console.log('========================');
console.log(`Total sets: ${LORCANA_SETS.length}`);
console.log(`Successful: ${results.successful}`);
console.log(`Failed: ${results.failed}`);
console.log(`Total cards imported: ${results.total}`);
console.log(`Success rate: ${((results.successful / LORCANA_SETS.length) * 100).toFixed(1)}%`);
// Save detailed results
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const filename = `lorcana-import-results-${timestamp}.json`;
fs.writeFileSync(filename, JSON.stringify(results, null, 2));
console.log(`\n📄 Detailed results saved to: ${filename}`);
return results;
}
// Run the import if this script is executed directly
if (import.meta.url === `file://${process.argv[1]}`) {
importAllLorcana()
.then(() => {
console.log('\n🎉 Lorcana import completed!');
process.exit(0);
})
.catch((error) => {
console.error('❌ Import failed:', error);
process.exit(1);
});
}
export { importAllLorcana, importLorcanaSet };