deckhearth/scripts/README.md

# Deck Hearth Bulk Import Scripts

This directory contains scripts for bulk importing TCG card data into the database.

## Available Scripts

### Card Import Scripts

#### 1. `import-popular-sets.js` - Popular Sets Import
Imports the most popular and recent sets from all three TCGs (Magic, Pokemon, Lorcana).

**Usage:**
```bash
npm run import-popular
```

**What it imports:**
- **Magic: The Gathering**: ~100+ sets from Alpha to recent releases
- **Pokemon**: ~100+ sets from Base Set to current Scarlet & Violet
- **Lorcana**: All 3 available sets

**Estimated time:** 2-4 hours (depending on API response times)

#### 2. `bulk-import-all.js` - Complete Import
Imports ALL available sets from all TCGs (comprehensive import).

**Usage:**
```bash
npm run import-all
```

**What it imports:**
- **Magic: The Gathering**: 100+ sets (Alpha to current)
- **Pokemon**: 100+ sets (Base Set to current)
- **Lorcana**: All available sets

**Estimated time:** 4-8 hours (depending on API response times)

### User Management Scripts

#### 3. `list-users.js` - List All Users
Lists all users in the database with their roles and details.

**Usage:**
```bash
node scripts/list-users.js
```

**Output:** Shows user ID, email, role (admin/user), and creation date.

#### 4. `promote-user-to-admin.js` - Promote User to Admin
Promotes a regular user to admin role.

**Usage:**
```bash
node scripts/promote-user-to-admin.js user@example.com
```

**Requirements:** User must be registered first.

#### 5. `demote-admin-to-user.js` - Demote Admin to User
Demotes an admin back to regular user role.

**Usage:**
```bash
node scripts/demote-admin-to-user.js admin@example.com
```

**Safety:** Ensures at least one admin always remains in the system.

### Database Management Scripts

#### 6. `setup-neon-db.js` - Database Setup
Sets up the Neon PostgreSQL database with all required tables and creates the default admin user.

**Usage:**
```bash
node scripts/setup-neon-db.js
```

#### 7. `reset-db.js` - Database Reset
Resets the database by dropping and recreating all tables.

**Usage:**
```bash
node scripts/reset-db.js
```

## How It Works

1. **Sequential Import**: Scripts import sets one by one to avoid overwhelming the APIs
2. **Error Handling**: Failed imports are logged but don't stop the process
3. **Progress Tracking**: Real-time console output shows progress
4. **Results Logging**: Detailed results are saved to JSON files
5. **Rate Limiting**: 1-second delays between imports to be respectful to APIs

## Output Files

After running, you'll get timestamped JSON files with detailed results:
- `popular-sets-import-results-[timestamp].json`
- `bulk-import-results-[timestamp].json`

These files contain:
- Success/failure status for each set
- Number of cards imported per set
- Error messages for failed imports
- Summary statistics

## Prerequisites

1. **Server Running**: Make sure your Next.js dev server is running (`npm run dev`)
2. **Database Setup**: Ensure the database is initialized (`npm run setup-db`)
3. **Dependencies**: All required packages are installed

## Recommendations

### For First-Time Setup
Start with the popular sets import:
```bash
npm run import-popular
```

This will give you a solid foundation with the most relevant cards.

### For Complete Database
If you want everything, use the full import:
```bash
npm run import-all
```

### For Ongoing Management
After the initial bulk import:

- **Weekly catalog sync (recommended):** Vercel Cron hits `GET /api/cron/sync-catalog`
  every Monday 06:00 UTC when `CRON_SECRET` is set in the Vercel project. The job
  discovers missing MTG + Pokémon sets and imports up to 3 per run (paced for upstream
  rate limits). Manual trigger:

  ```bash
  curl -H "Authorization: Bearer $CRON_SECRET" https://<your-deployment>/api/cron/sync-catalog
  ```

- **Admin UI:** `/admin/card-import` for one-off set imports (MTG + Pokémon).
- **Lorcana:** still manual via admin/scripts until dynamic set discovery lands.

Legacy manual path (still valid):
- Importing new sets as they release
- Selective imports of specific sets
- Monitoring import progress

## Troubleshooting

### Common Issues

1. **API Rate Limits**: If you get rate limit errors, the scripts will continue but log failures
2. **Network Issues**: Scripts will retry and continue with the next set
3. **Server Restart**: If the server restarts, just restart the import script

### Monitoring Progress

Watch the console output for:
- ✅ Successful imports with card counts
- ❌ Failed imports with error messages
- 📊 Summary statistics at the end

### Stopping and Resuming

You can stop the script with `Ctrl+C` and restart it later. The scripts will start from the beginning, but the database will only store unique cards (no duplicates).

## API Endpoints Used

- `POST /api/cards/import-mtg` - Magic: The Gathering imports
- `POST /api/cards/import-pokemon` - Pokemon imports  
- `POST /api/cards/import-lorcana` - Lorcana imports

## Data Sources

- **Magic: The Gathering**: Scryfall API
- **Pokemon**: [PokemonTCG/pokemon-tcg-data](https://github.com/PokemonTCG/pokemon-tcg-data) on GitHub (sets + per-set JSON; images from linked CDNs). Optional override: `POKEMON_TCG_DATA_BASE_URL` (defaults to `master` branch raw URLs). The legacy Pokemon TCG API is no longer used by catalog sync.
- **Lorcana**: Lorcana API (limited availability)

## Performance Notes

- Each set typically contains 100-400 cards
- Total database size after full import: ~50,000-100,000 cards
- Import speed: ~1 set per minute (with delays)
- Database storage: ~100-200MB after full import