Submatter/docs/deployment-production.md

# Echodo / Submatter — Production Deployment

Production runs on **CT 116** (`submatter`, `192.168.68.116`) as plain Docker Compose — not Dokploy.

| Doc | Purpose |
|-----|---------|
| This file | Operator checklist, env vars, troubleshooting |
| [`docs/forgejo-migration.md`](forgejo-migration.md) | GitHub → Forgejo + CT 116 cutover |
| [`proxmox/ct116/README.md`](../proxmox/ct116/README.md) | CT creation, Traefik, auto-deploy, Dokploy removal |
| `AGENT-DEPLOY.md` (local, gitignored) | Homelab credentials + CT map — add CT 116 row after cutover |

---

## Why a dedicated CT (not Dokploy)?

Echodo builds **five Docker images** from one monorepo; the Next.js compile alone reserves a 5 GB V8 heap. On Dokploy (CT 112), that competes with Zest, Umami, Comms, etc. and OOMs with exit 255.

**CT 116** gets the full 8 GB for serialized builds. CT 107 is not an alternative — it still hosts Coolify, Echo Board, Resonance, and other dev apps.

| Resource | CT 116 spec |
|----------|-------------|
| RAM | 8 GB |
| CPU | 2 cores |
| Disk | 32 GB rootfs |
| Runtime (Phase A) | ~3–4 GB |
| Phase B (+ OpenHands) | +2 GB |

External: Postgres/Redis on **CT 102**, TLS on **CT 100**, OmniRoute on **CT 108**.

---

## Stack files

| File | Use |
|------|-----|
| `docker/docker-compose.prod.yml` | Core stack |
| `docker/docker-compose.prod.standalone.yml` | Publish `:3000` / `:1234` for CT 100 Traefik |
| `scripts/build-prod.sh` | Serialized image build |
| `scripts/deploy-prod.sh` | Build + up + health wait |

```bash
docker compose -f docker/docker-compose.prod.yml \
  -f docker/docker-compose.prod.standalone.yml up -d
```

---

## Phase A — First deploy on CT 116

1. **Create CT 116** — see [`proxmox/ct116/README.md`](../proxmox/ct116/README.md) (`pct create` one-liner).
2. **Bootstrap** — `bootstrap.sh` installs Docker, clones to `/opt/submatter`.
3. **Configure `.env`** — copy from `.env.example`; use `echodo` database on CT 102, Redis `/11`.
4. **Deploy** — `bash scripts/deploy-prod.sh build`
5. **Traefik** — apply `proxmox/ct116/traefik-ct100.example.yml` on CT 100; remove old Dokploy/Coolify Echodo routes.
6. **Verify** — `https://echodo.stillwell.cloud`, `wss://collab.echodo.stillwell.cloud`

### Required env vars

| Variable | Production example | Notes |
|----------|-------------------|-------|
| `DATABASE_URL` | `postgresql://echodo:***@192.168.68.102:5432/echodo` | |
| `REDIS_URL` | `redis://:***@192.168.68.102:6379/11` | |
| `NEXTAUTH_URL` | `https://echodo.stillwell.cloud` | |
| `NEXTAUTH_SECRET` / `AUTH_SECRET` | `openssl rand -base64 32` | |
| `AUTH_AUTHENTIK_*` | | Authentik OIDC app `echodo` |
| `COLLAB_SERVER_URL` | `wss://collab.echodo.stillwell.cloud` | Server-side |
| `NEXT_PUBLIC_COLLAB_SERVER_URL` | `wss://collab.echodo.stillwell.cloud` | **Build arg** — requires image rebuild |
| `OMNIROUTE_BASE_URL` | `https://omniroute.stillwell.cloud/v1` | Web + worker |
| `OMNIROUTE_API_KEY` | (your key) | + register in Settings → AI → Providers |
| `FORGEJO_TOKEN` / `GITHUB_TOKEN` | PAT values | Referenced by name in Connected repos settings |
| `NEXT_PUBLIC_UMAMI_*` | | Analytics (CT 112 Umami) |

Changing any `NEXT_PUBLIC_*` value requires `bash scripts/deploy-prod.sh build`, not just `docker compose restart`.

---

## Auto-deploy on push

**CI** (every PR): `.github/workflows/ci.yml` — lint, type-check, test on GitHub-hosted or PR runner.

**CD** (merge to `main`): `.github/workflows/deploy-homelab.yml` and `.forgejo/workflows/deploy-homelab.yml` — SSH from CT 111 `axiom` runner → CT 116 → `deploy-prod.sh build`.

### GitHub / Forgejo repo secrets

| Secret | Value |
|--------|-------|
| `HOMELAB_DEPLOY_HOST` | `192.168.68.116` |
| `HOMELAB_DEPLOY_USER` | `root` |
| `HOMELAB_DEPLOY_SSH_KEY` | SSH private key (ed25519) |
| `HOMELAB_DEPLOY_PATH` | `/opt/submatter` (optional) |

Generate deploy key on CT 116:

```bash
ssh-keygen -t ed25519 -f /root/.ssh/github_deploy -N ""
cat /root/.ssh/github_deploy.pub >> /root/.ssh/authorized_keys
# Paste private key into HOMELAB_DEPLOY_SSH_KEY secret
```

Until secrets are set, the deploy workflow skips gracefully (CI still runs).

### Forgejo webhook (optional second trigger)

If Forgejo is canonical git and you want push deploy without Actions:

1. Install [webhook](https://github.com/adhocore/webhook) on CT 116.
2. Copy `proxmox/ct116/webhook.hooks.json.example` → `/etc/webhook.conf`.
3. Forgejo repo → Webhook → POST push events → `http://192.168.68.116:9000/hooks/submatter-deploy`.

---

## Remove from Dokploy

1. Dokploy UI → Submatter/Echodo compose app → Stop → Delete.
2. CT 100 — remove Traefik routes to Dokploy for `echodo.stillwell.cloud` / collab hostname.
3. Update local `AGENT-DEPLOY.md`: add CT 116 row, note Echodo removed from CT 112.

---

## Phase B — OpenCode (preferred)

```bash
export COMPOSE_PROFILES=opencode
# Set in .env: WORKER_SESSION_TYPE=opencode, OPENCODE_SERVER_PASSWORD, OMNIROUTE_*
bash scripts/deploy-prod.sh build   # first time / worker code change
# or: bash scripts/deploy-prod.sh up
```

| Variable | Notes |
|----------|-------|
| `OPENCODE_SERVER_PASSWORD` | `openssl rand -hex 24` — Traefik still fronts Authentik |
| `WORKER_SESSION_TYPE` | `opencode` |
| `OPENCODE_SERVER_URL` | `http://opencode:4096` (Docker DNS) |
| `OMNIROUTE_BASE_URL` | `https://omniroute.stillwell.cloud/v1` or LAN `http://192.168.68.108:20128/v1` |
| `OMNIROUTE_API_KEY` | OmniRoute bearer key |

Web UI: `https://opencode.stillwell.cloud`. Details: `docker/opencode/README.md`.

### Legacy — OpenHands (optional)

```bash
export COMPOSE_PROFILES=agent-server
bash scripts/deploy-prod.sh build
```

| Variable | Notes |
|----------|-------|
| `LOCAL_BACKEND_API_KEY` | `openssl rand -hex 32` |
| `SESSION_API_KEY` | Same as above (worker) |
| `WORKER_SESSION_TYPE` | `openhands` |

Requires Docker socket on CT 116 (`nesting=1` on LXC). Prefer OpenCode unless debugging the old path.
---

## Troubleshooting

| Symptom | Fix |
|---------|-----|
| Build exit 255 ~60–90s | Ensure `deploy-prod.sh` uses `build-prod.sh` (serialized) |
| OAuth broken | DB must be `echodo`, not legacy `tasks` |
| Deploy workflow skipped | Set `HOMELAB_DEPLOY_*` secrets |
| Collab websocket fails | Traefik route to `:1234`, `wss://` in public env vars |
| OpenHands unavailable | `SESSION_API_KEY` must match `LOCAL_BACKEND_API_KEY` |

Local build verify: `pnpm docker:build:prod`

---

## Future: registry pull deploy (no build on CT 116)

Build images on CT 111 CI, push to Forgejo container registry, then on CT 116:

```bash
bash scripts/deploy-prod.sh pull
```

Requires compose `image:` tags instead of `build:` — not implemented yet; track as follow-on if CT 116 builds remain too slow.

---

## Related

- `docs/AGENT-AUTONOMOUS-CONVOYS.md` — convoy + OpenHands roadmap
- `docs/omniroute-setup.md` — OmniRoute operator guide
- `.env.example` — full variable list