Commit graph

32 commits

Author SHA1 Message Date
Randall Stillwell
393114e9be Fix PDF page count detection and add processing diagnostics
The old getPdfPageCount used a fragile regex on raw PDF binary that
could undercount pages, causing survey sides to be skipped entirely.
Replaced with pdf-lib's PDFDocument.load() for reliable counting,
with the old regex as a fallback.

Added logging for page counts and pairing to diagnose missing backs.

Made-with: Cursor
2026-04-07 22:12:21 -05:00
Randall Stillwell
1d20932ef5 Add OCR rate limit handling and batch reprocess action
Rate limiting:
- Retry with exponential backoff (5s, 10s, 20s, 40s) on rate limit errors
- Reduce concurrent OCR jobs from 2 to 1
- Add 2s delay between AI calls within a card (response vs survey side)
- Add 3s delay between cards in a multi-page PDF
- Add 3s delay between jobs in the processing queue

Batch reprocess:
- New /api/cards/reprocess-batch endpoint processes cards sequentially
  with delays to respect rate limits
- Reprocess button added to selection toolbar on dashboard

Made-with: Cursor
2026-04-07 17:22:33 -05:00
Randall Stillwell
9b27041970 Fix Monday.com column type names: checkbox, status, numbers
Monday.com API returns column types as "checkbox" (not "boolean"),
"status" (not "color"), and "numbers" (not "numeric"). Added these
as aliases in formatForColumnType so values are properly formatted.

Made-with: Cursor
2026-04-07 17:09:59 -05:00
Randall Stillwell
d7d859f49d Fix Monday.com push: format column values by column type
The root cause was sending raw text to checkbox, status, dropdown, and
date columns. Monday.com requires specific JSON structures per type.

- Store column types in _columnTypes when Fetch Columns is clicked
- Format boolean/checkbox: {"checked": "true"/"false"}
- Format status/color: {"label": "value"}
- Format dropdown: {"labels": ["val1", "val2"]}
- Format date: parse MM/DD/YYYY to {"date": "YYYY-MM-DD"}
- Format email/phone/link with proper nested structures
- Text columns pass through as plain strings

Made-with: Cursor
2026-04-07 16:01:44 -05:00
Randall Stillwell
a0f5022de2 Fix Monday.com push failures and improve error visibility
- Fix toast showing success when all cards fail (now shows error/warning)
- Fix column value formatting: booleans use checkbox format, JSON arrays
  join as comma-separated strings, objects serialize properly
- Filter out empty/null/undefined string values before sending to API
- Add verbose logging to Monday.com GraphQL client for debugging

Made-with: Cursor
2026-04-07 13:50:44 -05:00
Randall Stillwell
034c73128f Add runtime safeguards: OCR concurrency limiter, upload size cap, and batched Monday sync
- Processing queue limits OCR to 2 concurrent jobs across all sources (upload, email, folder watcher)
- 50 MB file size cap enforced on uploads, email attachments, and folder watcher
- Monday.com bulk sync now processes in batches of 5 with 1s delay between batches

Made-with: Cursor
2026-04-07 13:44:04 -05:00
Randall Stillwell
7aefe7c4a1 Add manual push to Monday.com (single card + bulk sync)
- Extract reusable pushCardToMonday() that creates or updates Monday items
- New /api/integrations/monday/push/[id] endpoint for single-card push
- New /api/integrations/monday/sync-all endpoint for bulk push of all
  unsynced cards (ocrStatus=complete, no mondayItemId)
- Add "Push to Monday.com" button on card detail page header (shows
  "Update Monday" when card already has a Monday item)
- Add "Push All to Monday.com" button in Monday.com settings card

Made-with: Cursor
2026-04-07 13:32:15 -05:00
Randall Stillwell
2fcdd687f7 Fix duplex scan rotation and swap front/back image labels
Duplex scanners flip the sheet between sides, so odd pages (back) are
180° rotated relative to even pages (front). Landscape back pages now
rotate CCW instead of CW, and portrait back pages get a 180° flip.
Also swapped "Front"/"Back" labels to match the physical card sides.

Made-with: Cursor
2026-04-07 13:27:16 -05:00
Randall Stillwell
6cef79309b Auto-resume email and folder watchers on server startup
Added instrumentation.ts that reads persisted watcher state from the
database and restarts active watchers when the Next.js server boots.
Also added imapflow/mailparser to serverExternalPackages.

Made-with: Cursor
2026-04-07 13:14:41 -05:00
Randall Stillwell
d56b1a27ec Fix sideways scanned images by detecting and rotating landscape pages
Scanners store landscape content with a PDF /Rotate flag that pdf2pic
doesn't always apply. Now uses a square bounding box with
preserveAspectRatio, then detects landscape output and rotates to
portrait with sharp.

Made-with: Cursor
2026-04-07 13:10:03 -05:00
Randall Stillwell
f87ab36493 Fix Monday.com webhook subscription and improve API client
- Add API-Version header to all Monday.com GraphQL requests
- Switch create_webhook/delete_webhook to inline values matching Monday's
  documented syntax (fixes Internal Server Error on subscribe)
- Add extended error details from Monday.com GraphQL responses

Made-with: Cursor
2026-04-07 13:07:46 -05:00
Randall Stillwell
d58285785e Add Monday.com board fields and expand column mapper
- Add followUp, notes, serviceTime, planningCenter, iSaidYesBookSent,
  ftGuestLetterSent fields to ResponseCard to match Monday.com board
- Expand column mapper from 8 fields to all 32 ResponseCard fields with
  friendly labels matching the Easter Survey spreadsheet columns
- Add Workflow & Tracking section on card detail page for the new fields
- Fix pre-existing Buffer type error in images API route

Made-with: Cursor
2026-04-07 12:50:03 -05:00
Randall Stillwell
58f38d92ed Fix broken images by proxying through the app instead of redirecting to internal MinIO
The presigned URLs pointed to a private IP unreachable from the browser.
The image route now streams bytes directly and the card API returns
proxy paths matching the dashboard's existing approach.

Made-with: Cursor
2026-04-07 12:38:08 -05:00
Randall Stillwell
72375d2fae Add Integration Hub: Monday.com sync, webhooks, activity log, notifications
- Bidirectional Monday.com integration with column mapping and file uploads
- Generic webhook system with HMAC-SHA256 signing and configurable events
- Field-level activity log with diff tracking on card detail page
- Persistent notification center with unread badge in header
- Event dispatcher wired into OCR pipeline, card edits, exports, and deletes
- New Prisma models: ActivityLog, Notification; new fields on AppSettings and ResponseCard
- Settings UI with full Monday.com and webhook configuration panels

Made-with: Cursor
2026-04-07 12:23:29 -05:00
Randall Stillwell
6fdf68bc89 Fix profile dropdown crash by wrapping GroupLabel in Group
Menu.GroupLabel requires a parent Menu.Group for context. Without it,
clicking the profile avatar crashed the client.

Made-with: Cursor
2026-04-07 12:13:16 -05:00
Randall Stillwell
41000337e3 Add IMAP email monitoring for scanned card attachments
Watches an email inbox via IMAP IDLE for incoming scanned cards,
extracts PDF/image attachments, and feeds them through the existing
OCR pipeline. Configurable via the Settings page with test connection
support.

Made-with: Cursor
2026-04-07 11:41:52 -05:00
Randall Stillwell
3dd4dc16a3 Fix dropdown crash by removing submenu/group components
Replace DropdownMenuSub/DropdownMenuGroup with flat menu items
to avoid base-ui runtime errors. Theme now cycles with a single
click (Light → Dark → System).

Made-with: Cursor
2026-03-12 12:38:02 -05:00
Randall Stillwell
9ea9ca741c Allow reprocessing any card, not just errored ones
The Reprocess button and dropdown item were only visible for cards
with ocrStatus "error". Now they appear for any card that isn't
currently processing, so users can re-run OCR on completed cards too
(e.g. after changing AI settings).

Made-with: Cursor
2026-03-12 12:36:22 -05:00
Randall Stillwell
a6270a605a Add reprocess action to data table row dropdown
Cards with OCR error status now show a "Reprocess" option in the
row actions dropdown menu on the dashboard, triggering the existing
reprocess API endpoint.

Made-with: Cursor
2026-03-12 10:46:15 -05:00
Randall Stillwell
38f4229d3b Add profile page, user dropdown, and Authentik SSO integration
- Profile dropdown in header shows user name/email/avatar with links
  to profile, settings, support, theme submenu, and sign out
- New /profile page with editable user info (job title, company, bio)
  and read-only SSO fields when behind Authentik forward-auth
- /api/auth/me endpoint reads X-authentik-* headers and enriches
  with avatar from Authentik API
- Settings page gains Preferences tab with appearance theme picker
  and notification toggle switches
- User profile context backed by localStorage, merges with Authentik
  data when available

Made-with: Cursor
2026-03-12 10:42:39 -05:00
Randall Stillwell
59c4af5db6 Show placeholder text in filter dropdowns instead of __all__
Pass null instead of "__all__" as the Select value when no filter is
active, so the placeholder text (Visit Type, Attendance, Service) is
displayed by the base-ui SelectValue component.

Made-with: Cursor
2026-03-11 18:40:29 -05:00
Randall Stillwell
1168c96ae2 Fix right-click column visibility context menu crash
Replace base-ui DropdownMenu (requires anchor trigger) with a custom
positioned panel for the header right-click context menu. The base-ui
Menu primitive cannot be rendered free-floating without a trigger element.

Made-with: Cursor
2026-03-11 16:16:00 -05:00
Randall Stillwell
7ff9724a63 Unified dashboard redesign: Quicksand font, charcoal theme, merged upload
- Switch font from Geist to Google Quicksand
- Replace indigo/violet primary with charcoal gray (rgb 31,32,29)
- Update gradient-mesh to warm neutral tones (amber/sage)
- Remove sidebar, add fixed top header bar with profile dropdown
- Merge dashboard and upload pages into single table-centric view
- Stat cards now act as clickable table filters with active highlight
- Simplified filter toolbar: search + dropdowns left, export + upload right
- Upload modal with drag-drop, per-file XHR progress, background uploads
- Global drag-and-drop listener opens upload modal from anywhere
- Table rows are clickable (navigate to detail page)
- Right-click column headers for visibility context menu
- Mobile-responsive card layout below md breakpoint
- Uploading row placeholders with animated progress bars
- Floating selection toolbar slides in at bottom on multi-select
- Delete sidebar, dashboard-hero, quick-actions, upload page

Made-with: Cursor
2026-03-11 15:26:56 -05:00
Randall Stillwell
eabfb4935d Fix nullable Json field in reprocess and Prisma import path
Use Prisma.DbNull instead of null for the cardIds Json? field,
and correct the generated client import path.

Made-with: Cursor
2026-03-11 11:49:33 -05:00
Randall Stillwell
77b9fbea4c Fix AI test endpoint min token requirement
OpenAI requires maxOutputTokens >= 16; bump from 10 to 20.

Made-with: Cursor
2026-03-11 11:46:25 -05:00
Randall Stillwell
2db14b7336 Switch to Vercel AI Gateway for multi-provider OCR
Replace individual provider SDKs (@ai-sdk/openai, @ai-sdk/google,
@ai-sdk/anthropic) with the Vercel AI Gateway. Cloud models are now
accessed via a single AI_GATEWAY_API_KEY with provider/model strings
(e.g. openai/gpt-4o-mini, google/gemini-2.5-flash). Ollama remains
available as a local fallback via @ai-sdk/openai-compatible.

Made-with: Cursor
2026-03-11 11:05:38 -05:00
Randall Stillwell
1ec50d9033 Integrate Vercel AI SDK for multi-provider OCR
Replace the raw Ollama fetch-based OCR with the Vercel AI SDK,
adding support for OpenAI, Google Gemini, Anthropic, and Ollama
as selectable providers from the Settings page. Uses generateText
with Output.object() and Zod schemas for type-safe structured
data extraction.

Made-with: Cursor
2026-03-11 09:24:15 -05:00
Randall Stillwell
3018198196 Add /api/health endpoint to diagnose MinIO connectivity
Tests S3 bucket access and falls back to a raw HTTP fetch to show
what the endpoint actually returns (helps identify port misconfiguration).

Made-with: Cursor
2026-03-10 14:11:16 -05:00
Randall Stillwell
3b8d580ded Add database auto-migration, storage cleanup, and startup fixes
- Add start.sh that runs prisma db push before starting the server,
  creating tables automatically on first deploy
- Copy prisma CLI + engines into production image for runtime migrations
- Add configurable storage cleanup: source PDF retention (default 30d)
  and image retention (default 180d) with manual trigger in Settings
- Add /api/cleanup endpoint (GET for status, POST to run)
- Add retention settings to AppSettings schema
- Add ListObjectsV2 helper to minio.ts for bulk cleanup

Tested: local Docker build passes
Made-with: Cursor
2026-03-10 13:21:05 -05:00
Randall Stillwell
7e423a3e50 Build Echo OCR app: full-stack response card scanner
- Next.js 15 + Tailwind v4 + shadcn/ui + TanStack Table
- Prisma + PostgreSQL schema for ResponseCard, ProcessingJob, AppSettings
- Ollama vision integration with structured extraction prompts
- PDF-to-image pipeline with pdf2pic + sharp
- MinIO S3 storage for uploaded files and extracted images
- Chokidar-based folder monitoring for automatic processing
- REST API (9 endpoints) ready for Monday.com integration
- Dashboard with filterable data table, chip filters, bulk actions, CSV export
- Card detail page with editable fields and side-by-side scanned images
- Upload page with drag-and-drop and real-time processing queue
- Settings page for Ollama, folder watch, and Monday.com config
- Dockerfile (multi-stage) + docker-compose for local dev
- Configured for Coolify deployment at echo.stillwell.cloud

Made-with: Cursor
2026-03-10 07:17:45 -05:00
Randall Stillwell
6a6a140139 feat: initial commit 2026-03-09 15:31:45 -05:00
Randall Stillwell
95d05ac92e Initial commit from Create Next App 2026-03-09 15:31:21 -05:00