deckhearth/scripts/README.md

Deck Hearth Bulk Import Scripts

This directory contains scripts for bulk importing TCG card data into the database.

Available Scripts

Card Import Scripts

1. import-popular-sets.js - Popular Sets Import

Imports the most popular and recent sets from all three TCGs (Magic, Pokemon, Lorcana).

Usage:

npm run import-popular

What it imports:

• Magic: The Gathering: ~100+ sets from Alpha to recent releases
• Pokemon: ~100+ sets from Base Set to current Scarlet & Violet
• Lorcana: All 3 available sets

Estimated time: 2-4 hours (depending on API response times)

2. bulk-import-all.js - Complete Import

Imports ALL available sets from all TCGs (comprehensive import).

Usage:

npm run import-all

What it imports:

• Magic: The Gathering: 100+ sets (Alpha to current)
• Pokemon: 100+ sets (Base Set to current)
• Lorcana: All available sets

Estimated time: 4-8 hours (depending on API response times)

User Management Scripts

3. list-users.js - List All Users

Lists all users in the database with their roles and details.

Usage:

node scripts/list-users.js

Output: Shows user ID, email, role (admin/user), and creation date.

4. promote-user-to-admin.js - Promote User to Admin

Promotes a regular user to admin role.

Usage:

node scripts/promote-user-to-admin.js user@example.com

Requirements: User must be registered first.

5. demote-admin-to-user.js - Demote Admin to User

Demotes an admin back to regular user role.

Usage:

node scripts/demote-admin-to-user.js admin@example.com

Safety: Ensures at least one admin always remains in the system.

Database Management Scripts

6. setup-neon-db.js - Database Setup

Sets up the Neon PostgreSQL database with all required tables and creates the default admin user.

Usage:

node scripts/setup-neon-db.js

7. reset-db.js - Database Reset

Resets the database by dropping and recreating all tables.

Usage:

node scripts/reset-db.js

How It Works

1. Sequential Import: Scripts import sets one by one to avoid overwhelming the APIs
2. Error Handling: Failed imports are logged but don't stop the process
3. Progress Tracking: Real-time console output shows progress
4. Results Logging: Detailed results are saved to JSON files
5. Rate Limiting: 1-second delays between imports to be respectful to APIs

Output Files

After running, you'll get timestamped JSON files with detailed results:

• popular-sets-import-results-[timestamp].json
• bulk-import-results-[timestamp].json

These files contain:

• Success/failure status for each set
• Number of cards imported per set
• Error messages for failed imports
• Summary statistics

Prerequisites

1. Server Running: Make sure your Next.js dev server is running (npm run dev)
2. Database Setup: Ensure the database is initialized (npm run setup-db)
3. Dependencies: All required packages are installed

Recommendations

For First-Time Setup

Start with the popular sets import:

npm run import-popular

This will give you a solid foundation with the most relevant cards.

For Complete Database

If you want everything, use the full import:

npm run import-all

For Ongoing Management

After the initial bulk import:

• Weekly catalog sync (recommended): Vercel Cron hits GET /api/cron/sync-catalog every Monday 06:00 UTC when CRON_SECRET is set in the Vercel project. The job discovers missing MTG + Pokémon sets and imports up to 3 per run (paced for upstream rate limits). Manual trigger:

  curl -H "Authorization: Bearer $CRON_SECRET" https://<your-deployment>/api/cron/sync-catalog
  

• Admin UI: /admin/card-import for one-off set imports (MTG + Pokémon).

• Lorcana: still manual via admin/scripts until dynamic set discovery lands.

Legacy manual path (still valid):

• Importing new sets as they release
• Selective imports of specific sets
• Monitoring import progress

Troubleshooting

Common Issues

1. API Rate Limits: If you get rate limit errors, the scripts will continue but log failures
2. Network Issues: Scripts will retry and continue with the next set
3. Server Restart: If the server restarts, just restart the import script

Monitoring Progress

Watch the console output for:

• ✅ Successful imports with card counts
• ❌ Failed imports with error messages
• 📊 Summary statistics at the end

Stopping and Resuming

You can stop the script with Ctrl+C and restart it later. The scripts will start from the beginning, but the database will only store unique cards (no duplicates).

API Endpoints Used

• POST /api/cards/import-mtg - Magic: The Gathering imports
• POST /api/cards/import-pokemon - Pokemon imports
• POST /api/cards/import-lorcana - Lorcana imports

Data Sources

• Magic: The Gathering: Scryfall API
• Pokemon: Pokemon TCG API
• Lorcana: Lorcana API (limited availability)

Performance Notes

• Each set typically contains 100-400 cards
• Total database size after full import: ~50,000-100,000 cards
• Import speed: ~1 set per minute (with delays)
• Database storage: ~100-200MB after full import