Submatter/docs/deployment-production.md

Echodo / Submatter — Production Deployment

Production runs on CT 116 (submatter, 192.168.68.116) as plain Docker Compose — not Dokploy.

Doc	Purpose
This file	Operator checklist, env vars, troubleshooting
docs/forgejo-migration.md	GitHub → Forgejo + CT 116 cutover
proxmox/ct116/README.md	CT creation, Traefik, auto-deploy, Dokploy removal
AGENT-DEPLOY.md (local, gitignored)	Homelab credentials + CT map — add CT 116 row after cutover

---

Why a dedicated CT (not Dokploy)?

Echodo builds five Docker images from one monorepo; the Next.js compile alone reserves a 5 GB V8 heap. On Dokploy (CT 112), that competes with Zest, Umami, Comms, etc. and OOMs with exit 255.

CT 116 gets the full 8 GB for serialized builds. CT 107 is not an alternative — it still hosts Coolify, Echo Board, Resonance, and other dev apps.

Resource	CT 116 spec
RAM	8 GB
CPU	2 cores
Disk	32 GB rootfs
Runtime (Phase A)	~3–4 GB
Phase B (+ OpenHands)	+2 GB

External: Postgres/Redis on CT 102, TLS on CT 100, OmniRoute on CT 108.

---

Stack files

File	Use
docker/docker-compose.prod.yml	Core stack
docker/docker-compose.prod.standalone.yml	Publish :3000 / :1234 / :1235 for CT 100 Traefik
scripts/build-prod.sh	Serialized image build
scripts/deploy-prod.sh	Build + up + health wait

docker compose -f docker/docker-compose.prod.yml \
  -f docker/docker-compose.prod.standalone.yml up -d

---

Phase A — First deploy on CT 116

1. Create CT 116 — see proxmox/ct116/README.md (pct create one-liner).
2. Bootstrap — bootstrap.sh installs Docker, clones to /opt/submatter.
3. Configure .env — copy from .env.example; use echodo database on CT 102, Redis /11.
4. Deploy — bash scripts/deploy-prod.sh build
5. Traefik — apply proxmox/ct116/traefik-ct100.example.yml on CT 100; remove old Dokploy/Coolify Echodo routes.
6. Verify — https://echodo.stillwell.cloud, wss://collab.echodo.stillwell.cloud

Required env vars

Variable	Production example	Notes
DATABASE_URL	postgresql://echodo:***@192.168.68.102:5432/echodo
REDIS_URL	redis://:***@192.168.68.102:6379/11
NEXTAUTH_URL	https://echodo.stillwell.cloud
NEXTAUTH_SECRET / AUTH_SECRET	openssl rand -base64 32
AUTH_AUTHENTIK_*		Authentik OIDC app echodo
COLLAB_SERVER_URL	wss://collab.echodo.stillwell.cloud	Server-side
NEXT_PUBLIC_COLLAB_SERVER_URL	wss://collab.echodo.stillwell.cloud	Build arg — requires image rebuild
NEXT_PUBLIC_CHAT_SERVER_URL	wss://echodo-chat.stillwell.cloud	Build arg — requires web image rebuild; provision DNS/TLS first
CHAT_TICKET_SECRET	Generate with openssl rand -base64 32	Same runtime secret for web and collab; at least 32 characters; never a build arg
CHAT_PORT	1235	Second listener inside collab; cannot equal Hocuspocus port 1234
OMNIROUTE_BASE_URL	https://omniroute.stillwell.cloud/v1	Web + worker
OMNIROUTE_API_KEY	(your key)	+ register in Settings → AI → Providers
FORGEJO_TOKEN / GITHUB_TOKEN	PAT values	Referenced by name in Connected repos settings
NEXT_PUBLIC_UMAMI_*		Analytics (CT 112 Umami)

Changing any NEXT_PUBLIC_* value requires bash scripts/deploy-prod.sh build, not just docker compose restart.

---

Workspace Chat integration gate

Chat runs inside the existing collab service on a separate listener from Hocuspocus. The worker publishes metadata-only Redis hints; Postgres history and mutation catch-up remain authoritative. Team Chat does not invoke AI agents, and object contexts are not enabled by this deployment.

1. Set the Chat variables above in /opt/submatter/.env. Keep the signing secret server-side; Compose requires a nonempty value, and the application requires at least 32 characters. A missing/short secret disables the listener and makes the dual-listener health check fail.

2. Provision echodo-chat.stillwell.cloud DNS and TLS on CT 100 (canonical browser endpoint). chat.stillwell.cloud belongs to another app: never include it in the Workspace Chat router. Apply the echodo-chat router/service from traefik-ct100.example.yml. Traefik supports WebSocket upgrades without a separate service process. If CHAT_PORT changes, update the proxy backend port too. Restrict host backend ports to the trusted proxy network.

   For the hostname migration, set NEXT_PUBLIC_CHAT_SERVER_URL=wss://echodo-chat.stillwell.cloud in CT 116's /opt/submatter/.env, then run from /opt/submatter:

   docker compose --project-directory /opt/submatter -f docker/docker-compose.prod.yml -f docker/docker-compose.prod.standalone.yml build web
   docker compose --project-directory /opt/submatter -f docker/docker-compose.prod.yml -f docker/docker-compose.prod.standalone.yml up -d --no-deps web
   

   A git push or container restart alone does not change the baked browser URL. Test Workspace Chat inside https://echodo.stillwell.cloud; its WebSocket should connect to wss://echodo-chat.stillwell.cloud/chat. The WSS endpoint is not a separate chat UI.

3. Before rolling out, verify the database target is CT 102 echodo and inspect drizzle.__drizzle_migrations. The operator reports 24 ledger entries through 0023, verified Chat schema, and ownership of all chat_* tables transferred to echodo. Do not reapply these migrations; inspect the ledger for any future pending entries. Compare the ledger with packages/database/migrations/meta/_journal.json; apply only pending journaled migrations using pnpm --filter @tasks/database db:migrate in a checkout with the real DATABASE_URL exported securely. Do not print the URL, use db:push, or replay historical migrations manually. Take a backup first.

4. Verify 0022 added chat_messages.send_intent_fingerprint and positive revision; verify 0023 added chat_message_mutations, its tenant/channel/message foreign keys and indexes, and chat_channels.next_mutation_sequence. Do not infer applied schema from TypeScript files. The historical 0017 correction is already local; do not replay it on an initialized database.

5. Validate Compose on the deployment host without printing resolved secrets:

   docker compose --project-directory /opt/submatter --env-file /opt/submatter/.env \
     -f docker/docker-compose.prod.yml -f docker/docker-compose.prod.standalone.yml config --quiet
   bash scripts/deploy-prod.sh build
   

6. Confirm both collab listeners and worker health. The TCP health check verifies listener liveness only, not Redis delivery, database authorization, or end-to-end readiness. An ordinary HTTP request to the Chat listener returns 404; use an authenticated browser WebSocket session for acceptance.

7. Test two signed-in workspace members: send/reply/edit/delete, unread acknowledgement, reconnect catch-up, offline edits/deletes, and duplicate retries. Revoke private/direct membership while connected and verify subsequent hints/history are denied and cached content is cleared; admins must not bypass membership. Verify another workspace cannot obtain history or subscribe. Restart collab/worker and test Redis outage recovery without message loss.

Run PostgreSQL integration tests only against a disposable test database, not production. DNS, TLS, migrations, Redis/WebSocket delivery, and two-browser acceptance are independent release gates; passing unit tests does not satisfy them. See Current Progress for the current evidence and blockers.

Post-go-live reconciliation

The operator reports Chat live with 14/14 acceptance checks; see Current Progress for provenance and scope. Rebuild web on the next deploy to replace hot-patched image chunks with source-built code. The reconciled ticket route uses AUTH_URL or NEXTAUTH_URL as its trusted public origin; set it to the HTTPS web app origin, not the WSS endpoint. It ignores forwarded headers for this decision and fails closed without valid production configuration. Restrict direct backend access to the proxy network regardless.

AUTH_DEV_PASSWORD was enabled for acceptance. This shared password can authenticate existing email identities; it is not individual-user password authentication. Confirm SSO works, then remove it and recreate web if it was temporary. Do not leave this enabled solely because acceptance needed it. Re-run ticket origin/authentication and live Chat acceptance after the source-built deployment.

Auto-deploy on push

CI (every PR): .github/workflows/ci.yml — lint, type-check, test on GitHub-hosted or PR runner.

CD (merge to main): .github/workflows/deploy-homelab.yml and .forgejo/workflows/deploy-homelab.yml — SSH from CT 111 axiom runner → CT 116 → deploy-prod.sh build.

GitHub / Forgejo repo secrets

Secret	Value
HOMELAB_DEPLOY_HOST	192.168.68.116
HOMELAB_DEPLOY_USER	root
HOMELAB_DEPLOY_SSH_KEY	SSH private key (ed25519)
HOMELAB_DEPLOY_PATH	/opt/submatter (optional)

Generate deploy key on CT 116:

ssh-keygen -t ed25519 -f /root/.ssh/github_deploy -N ""
cat /root/.ssh/github_deploy.pub >> /root/.ssh/authorized_keys
# Paste private key into HOMELAB_DEPLOY_SSH_KEY secret

Until secrets are set, the deploy workflow skips gracefully (CI still runs).

Forgejo webhook (optional second trigger)

If Forgejo is canonical git and you want push deploy without Actions:

1. Install webhook on CT 116.
2. Copy proxmox/ct116/webhook.hooks.json.example → /etc/webhook.conf.
3. Forgejo repo → Webhook → POST push events → http://192.168.68.116:9000/hooks/submatter-deploy.

---

Remove from Dokploy

1. Dokploy UI → Submatter/Echodo compose app → Stop → Delete.
2. CT 100 — remove Traefik routes to Dokploy for echodo.stillwell.cloud / collab hostname.
3. Update local AGENT-DEPLOY.md: add CT 116 row, note Echodo removed from CT 112.

---

Phase B — OpenCode (preferred)

export COMPOSE_PROFILES=opencode
# Set in .env: WORKER_SESSION_TYPE=opencode, OPENCODE_SERVER_PASSWORD, OMNIROUTE_*
bash scripts/deploy-prod.sh build   # first time / worker code change
# or: bash scripts/deploy-prod.sh up

Variable	Notes
OPENCODE_SERVER_PASSWORD	openssl rand -hex 24 — Traefik still fronts Authentik
WORKER_SESSION_TYPE	opencode
OPENCODE_SERVER_URL	http://opencode:4096 (Docker DNS)
OMNIROUTE_BASE_URL	https://omniroute.stillwell.cloud/v1 or LAN http://192.168.68.108:20128/v1
OMNIROUTE_API_KEY	OmniRoute bearer key

Web UI: https://opencode.stillwell.cloud. Details: docker/opencode/README.md.

Legacy — OpenHands (optional)

export COMPOSE_PROFILES=agent-server
bash scripts/deploy-prod.sh build

Variable	Notes
LOCAL_BACKEND_API_KEY	openssl rand -hex 32
SESSION_API_KEY	Same as above (worker)
WORKER_SESSION_TYPE	openhands

Requires Docker socket on CT 116 (nesting=1 on LXC). Prefer OpenCode unless debugging the old path.

Troubleshooting

Symptom	Fix
Build exit 255 ~60–90s	Ensure deploy-prod.sh uses build-prod.sh (serialized)
OAuth broken	DB must be echodo, not legacy tasks
Deploy workflow skipped	Set HOMELAB_DEPLOY_* secrets
Collab websocket fails	Traefik route to :1234, wss:// in public env vars
OpenHands unavailable	SESSION_API_KEY must match LOCAL_BACKEND_API_KEY

Local build verify: pnpm docker:build:prod

---

Future: registry pull deploy (no build on CT 116)

Build images on CT 111 CI, push to Forgejo container registry, then on CT 116:

bash scripts/deploy-prod.sh pull

Requires compose image: tags instead of build: — not implemented yet; track as follow-on if CT 116 builds remain too slow.

---

Related

• docs/AGENT-AUTONOMOUS-CONVOYS.md — convoy + OpenHands roadmap
• docs/omniroute-setup.md — OmniRoute operator guide
• .env.example — full variable list