Submatter/docs/deployment-production.md

Echodo / Submatter — Production Deployment

Production runs on CT 116 (submatter, 192.168.68.116) as plain Docker Compose — not Dokploy.

Doc	Purpose
This file	Operator checklist, env vars, troubleshooting
docs/forgejo-migration.md	GitHub → Forgejo + CT 116 cutover
proxmox/ct116/README.md	CT creation, Traefik, auto-deploy, Dokploy removal
AGENT-DEPLOY.md (local, gitignored)	Homelab credentials + CT map — add CT 116 row after cutover

---

Why a dedicated CT (not Dokploy)?

Echodo builds five Docker images from one monorepo; the Next.js compile alone reserves a 5 GB V8 heap. On Dokploy (CT 112), that competes with Zest, Umami, Comms, etc. and OOMs with exit 255.

CT 116 gets the full 8 GB for serialized builds. CT 107 is not an alternative — it still hosts Coolify, Echo Board, Resonance, and other dev apps.

Resource	CT 116 spec
RAM	8 GB
CPU	2 cores
Disk	32 GB rootfs
Runtime (Phase A)	~3–4 GB
Phase B (+ OpenHands)	+2 GB

External: Postgres/Redis on CT 102, TLS on CT 100, OmniRoute on CT 108.

---

Stack files

File	Use
docker/docker-compose.prod.yml	Core stack
docker/docker-compose.prod.standalone.yml	Publish :3000 / :1234 for CT 100 Traefik
scripts/build-prod.sh	Serialized image build
scripts/deploy-prod.sh	Build + up + health wait

docker compose -f docker/docker-compose.prod.yml \
  -f docker/docker-compose.prod.standalone.yml up -d

---

Phase A — First deploy on CT 116

1. Create CT 116 — see proxmox/ct116/README.md (pct create one-liner).
2. Bootstrap — bootstrap.sh installs Docker, clones to /opt/submatter.
3. Configure .env — copy from .env.example; use echodo database on CT 102, Redis /11.
4. Deploy — bash scripts/deploy-prod.sh build
5. Traefik — apply proxmox/ct116/traefik-ct100.example.yml on CT 100; remove old Dokploy/Coolify Echodo routes.
6. Verify — https://echodo.stillwell.cloud, wss://collab.echodo.stillwell.cloud

Required env vars

Variable	Production example	Notes
DATABASE_URL	postgresql://echodo:***@192.168.68.102:5432/echodo
REDIS_URL	redis://:***@192.168.68.102:6379/11
NEXTAUTH_URL	https://echodo.stillwell.cloud
NEXTAUTH_SECRET / AUTH_SECRET	openssl rand -base64 32
AUTH_AUTHENTIK_*		Authentik OIDC app echodo
COLLAB_SERVER_URL	wss://collab.echodo.stillwell.cloud	Server-side
NEXT_PUBLIC_COLLAB_SERVER_URL	wss://collab.echodo.stillwell.cloud	Build arg — requires image rebuild
OMNIROUTE_BASE_URL	https://omniroute.stillwell.cloud/v1	Web + worker
OMNIROUTE_API_KEY	(your key)	+ register in Settings → AI → Providers
FORGEJO_TOKEN / GITHUB_TOKEN	PAT values	Referenced by name in Connected repos settings
NEXT_PUBLIC_UMAMI_*		Analytics (CT 112 Umami)

Changing any NEXT_PUBLIC_* value requires bash scripts/deploy-prod.sh build, not just docker compose restart.

---

Auto-deploy on push

CI (every PR): .github/workflows/ci.yml — lint, type-check, test on GitHub-hosted or PR runner.

CD (merge to main): .github/workflows/deploy-homelab.yml and .forgejo/workflows/deploy-homelab.yml — SSH from CT 111 axiom runner → CT 116 → deploy-prod.sh build.

GitHub / Forgejo repo secrets

Secret	Value
HOMELAB_DEPLOY_HOST	192.168.68.116
HOMELAB_DEPLOY_USER	root
HOMELAB_DEPLOY_SSH_KEY	SSH private key (ed25519)
HOMELAB_DEPLOY_PATH	/opt/submatter (optional)

Generate deploy key on CT 116:

ssh-keygen -t ed25519 -f /root/.ssh/github_deploy -N ""
cat /root/.ssh/github_deploy.pub >> /root/.ssh/authorized_keys
# Paste private key into HOMELAB_DEPLOY_SSH_KEY secret

Until secrets are set, the deploy workflow skips gracefully (CI still runs).

Forgejo webhook (optional second trigger)

If Forgejo is canonical git and you want push deploy without Actions:

1. Install webhook on CT 116.
2. Copy proxmox/ct116/webhook.hooks.json.example → /etc/webhook.conf.
3. Forgejo repo → Webhook → POST push events → http://192.168.68.116:9000/hooks/submatter-deploy.

---

Remove from Dokploy

1. Dokploy UI → Submatter/Echodo compose app → Stop → Delete.
2. CT 100 — remove Traefik routes to Dokploy for echodo.stillwell.cloud / collab hostname.
3. Update local AGENT-DEPLOY.md: add CT 116 row, note Echodo removed from CT 112.

---

Phase B — OpenCode (preferred)

export COMPOSE_PROFILES=opencode
# Set in .env: WORKER_SESSION_TYPE=opencode, OPENCODE_SERVER_PASSWORD, OMNIROUTE_*
bash scripts/deploy-prod.sh build   # first time / worker code change
# or: bash scripts/deploy-prod.sh up

Variable	Notes
OPENCODE_SERVER_PASSWORD	openssl rand -hex 24 — Traefik still fronts Authentik
WORKER_SESSION_TYPE	opencode
OPENCODE_SERVER_URL	http://opencode:4096 (Docker DNS)
OMNIROUTE_BASE_URL	https://omniroute.stillwell.cloud/v1 or LAN http://192.168.68.108:20128/v1
OMNIROUTE_API_KEY	OmniRoute bearer key

Web UI: https://opencode.stillwell.cloud. Details: docker/opencode/README.md.

Legacy — OpenHands (optional)

export COMPOSE_PROFILES=agent-server
bash scripts/deploy-prod.sh build

Variable	Notes
LOCAL_BACKEND_API_KEY	openssl rand -hex 32
SESSION_API_KEY	Same as above (worker)
WORKER_SESSION_TYPE	openhands

Requires Docker socket on CT 116 (nesting=1 on LXC). Prefer OpenCode unless debugging the old path.

Troubleshooting

Symptom	Fix
Build exit 255 ~60–90s	Ensure deploy-prod.sh uses build-prod.sh (serialized)
OAuth broken	DB must be echodo, not legacy tasks
Deploy workflow skipped	Set HOMELAB_DEPLOY_* secrets
Collab websocket fails	Traefik route to :1234, wss:// in public env vars
OpenHands unavailable	SESSION_API_KEY must match LOCAL_BACKEND_API_KEY

Local build verify: pnpm docker:build:prod

---

Future: registry pull deploy (no build on CT 116)

Build images on CT 111 CI, push to Forgejo container registry, then on CT 116:

bash scripts/deploy-prod.sh pull

Requires compose image: tags instead of build: — not implemented yet; track as follow-on if CT 116 builds remain too slow.

---

Related

• docs/AGENT-AUTONOMOUS-CONVOYS.md — convoy + OpenHands roadmap
• docs/omniroute-setup.md — OmniRoute operator guide
• .env.example — full variable list