Submatter/docs/deployment-production.md

# Echodo / Submatter — Production Deployment

Production runs on **CT 116** (`submatter`, `192.168.68.116`) as plain Docker Compose — not Dokploy.

| Doc | Purpose |
|-----|---------|
| This file | Operator checklist, env vars, troubleshooting |
| [`docs/forgejo-migration.md`](forgejo-migration.md) | GitHub → Forgejo + CT 116 cutover |
| [`proxmox/ct116/README.md`](../proxmox/ct116/README.md) | CT creation, Traefik, auto-deploy, Dokploy removal |
| `AGENT-DEPLOY.md` (local, gitignored) | Homelab credentials + CT map — add CT 116 row after cutover |

---

## Why a dedicated CT (not Dokploy)?

Echodo builds **five Docker images** from one monorepo; the Next.js compile alone reserves a 5 GB V8 heap. On Dokploy (CT 112), that competes with Zest, Umami, Comms, etc. and OOMs with exit 255.

**CT 116** gets the full 8 GB for serialized builds. CT 107 is not an alternative — it still hosts Coolify, Echo Board, Resonance, and other dev apps.

| Resource | CT 116 spec |
|----------|-------------|
| RAM | 8 GB |
| CPU | 2 cores |
| Disk | 32 GB rootfs |
| Runtime (Phase A) | ~3–4 GB |
| Phase B (+ OpenHands) | +2 GB |

External: Postgres/Redis on **CT 102**, TLS on **CT 100**, OmniRoute on **CT 108**.

---

## Stack files

| File | Use |
|------|-----|
| `docker/docker-compose.prod.yml` | Core stack |
| `docker/docker-compose.prod.standalone.yml` | Publish `:3000` / `:1234` / `:1235` for CT 100 Traefik |
| `scripts/build-prod.sh` | Serialized image build |
| `scripts/deploy-prod.sh` | Build + up + health wait |

```bash
docker compose -f docker/docker-compose.prod.yml \
  -f docker/docker-compose.prod.standalone.yml up -d
```

---

## Phase A — First deploy on CT 116

1. **Create CT 116** — see [`proxmox/ct116/README.md`](../proxmox/ct116/README.md) (`pct create` one-liner).
2. **Bootstrap** — `bootstrap.sh` installs Docker, clones to `/opt/submatter`.
3. **Configure `.env`** — copy from `.env.example`; use `echodo` database on CT 102, Redis `/11`.
4. **Deploy** — `bash scripts/deploy-prod.sh build`
5. **Traefik** — apply `proxmox/ct116/traefik-ct100.example.yml` on CT 100; remove old Dokploy/Coolify Echodo routes.
6. **Verify** — `https://echodo.stillwell.cloud`, `wss://collab.echodo.stillwell.cloud`

### Required env vars

| Variable | Production example | Notes |
|----------|-------------------|-------|
| `DATABASE_URL` | `postgresql://echodo:***@192.168.68.102:5432/echodo` | |
| `REDIS_URL` | `redis://:***@192.168.68.102:6379/11` | |
| `NEXTAUTH_URL` | `https://echodo.stillwell.cloud` | |
| `NEXTAUTH_SECRET` / `AUTH_SECRET` | `openssl rand -base64 32` | |
| `AUTH_AUTHENTIK_*` | | Authentik OIDC app `echodo` |
| `COLLAB_SERVER_URL` | `wss://collab.echodo.stillwell.cloud` | Server-side |
| `NEXT_PUBLIC_COLLAB_SERVER_URL` | `wss://collab.echodo.stillwell.cloud` | **Build arg** — requires image rebuild |
| `NEXT_PUBLIC_CHAT_SERVER_URL` | `wss://chat.stillwell.cloud` | **Build arg** — requires web image rebuild; provision DNS/TLS first |
| `CHAT_TICKET_SECRET` | Generate with `openssl rand -base64 32` | Same runtime secret for web and collab; at least 32 characters; never a build arg |
| `CHAT_PORT` | `1235` | Second listener inside collab; cannot equal Hocuspocus port `1234` |
| `OMNIROUTE_BASE_URL` | `https://omniroute.stillwell.cloud/v1` | Web + worker |
| `OMNIROUTE_API_KEY` | (your key) | + register in Settings → AI → Providers |
| `FORGEJO_TOKEN` / `GITHUB_TOKEN` | PAT values | Referenced by name in Connected repos settings |
| `NEXT_PUBLIC_UMAMI_*` | | Analytics (CT 112 Umami) |

Changing any `NEXT_PUBLIC_*` value requires `bash scripts/deploy-prod.sh build`, not just `docker compose restart`.

---

## Workspace Chat integration gate

Chat runs inside the existing `collab` service on a **separate listener** from Hocuspocus. The worker publishes metadata-only Redis hints; Postgres history and mutation catch-up remain authoritative. Team Chat does not invoke AI agents, and object contexts are not enabled by this deployment.

1. Set the Chat variables above in `/opt/submatter/.env`. Keep the signing secret server-side; Compose requires a nonempty value, and the application requires at least 32 characters. A missing/short secret disables the listener and makes the dual-listener health check fail.
2. Provision `chat.stillwell.cloud` DNS and TLS on CT 100 (canonical browser endpoint). The proxy example also retains `chat.echodo.stillwell.cloud`; a single-level wildcard certificate does not cover that nested hostname. Apply the `echodo-chat` router/service from [`traefik-ct100.example.yml`](../proxmox/ct116/traefik-ct100.example.yml). Traefik supports WebSocket upgrades without a separate service process. If `CHAT_PORT` changes, update the proxy backend port too. Restrict host backend ports to the trusted proxy network.
3. Before rolling out, verify the database target is CT 102 **`echodo`** and inspect `drizzle.__drizzle_migrations`. The operator reports 24 ledger entries through `0023`, verified Chat schema, and ownership of all `chat_*` tables transferred to `echodo`. Do not reapply these migrations; inspect the ledger for any future pending entries. Compare the ledger with `packages/database/migrations/meta/_journal.json`; apply only pending journaled migrations using `pnpm --filter @tasks/database db:migrate` in a checkout with the real `DATABASE_URL` exported securely. Do not print the URL, use `db:push`, or replay historical migrations manually. Take a backup first.
4. Verify `0022` added `chat_messages.send_intent_fingerprint` and positive `revision`; verify `0023` added `chat_message_mutations`, its tenant/channel/message foreign keys and indexes, and `chat_channels.next_mutation_sequence`. Do not infer applied schema from TypeScript files. The historical `0017` correction is already local; do not replay it on an initialized database.
5. Validate Compose on the deployment host without printing resolved secrets:

   ```bash
   docker compose --project-directory /opt/submatter --env-file /opt/submatter/.env \
     -f docker/docker-compose.prod.yml -f docker/docker-compose.prod.standalone.yml config --quiet
   bash scripts/deploy-prod.sh build
   ```

6. Confirm both collab listeners and worker health. The TCP health check verifies listener liveness only, **not** Redis delivery, database authorization, or end-to-end readiness. An ordinary HTTP request to the Chat listener returns 404; use an authenticated browser WebSocket session for acceptance.
7. Test two signed-in workspace members: send/reply/edit/delete, unread acknowledgement, reconnect catch-up, offline edits/deletes, and duplicate retries. Revoke private/direct membership while connected and verify subsequent hints/history are denied and cached content is cleared; admins must not bypass membership. Verify another workspace cannot obtain history or subscribe. Restart collab/worker and test Redis outage recovery without message loss.

Run PostgreSQL integration tests only against a disposable test database, not production. DNS, TLS, migrations, Redis/WebSocket delivery, and two-browser acceptance are independent release gates; passing unit tests does not satisfy them. See [Current Progress](Current%20Progress.md) for the current evidence and blockers.

### Post-go-live reconciliation

The operator reports Chat live with 14/14 acceptance checks; see [Current Progress](Current%20Progress.md) for provenance and scope. Rebuild web on the next deploy to replace hot-patched image chunks with source-built code. The reconciled ticket route uses `AUTH_URL` or `NEXTAUTH_URL` as its trusted public origin; set it to the HTTPS **web app** origin, not the WSS endpoint. It ignores forwarded headers for this decision and fails closed without valid production configuration. Restrict direct backend access to the proxy network regardless.

`AUTH_DEV_PASSWORD` was enabled for acceptance. This shared password can authenticate existing email identities; it is not individual-user password authentication. Confirm SSO works, then remove it and recreate web if it was temporary. Do not leave this enabled solely because acceptance needed it. Re-run ticket origin/authentication and live Chat acceptance after the source-built deployment.

## Auto-deploy on push

**CI** (every PR): `.github/workflows/ci.yml` — lint, type-check, test on GitHub-hosted or PR runner.

**CD** (merge to `main`): `.github/workflows/deploy-homelab.yml` and `.forgejo/workflows/deploy-homelab.yml` — SSH from CT 111 `axiom` runner → CT 116 → `deploy-prod.sh build`.

### GitHub / Forgejo repo secrets

| Secret | Value |
|--------|-------|
| `HOMELAB_DEPLOY_HOST` | `192.168.68.116` |
| `HOMELAB_DEPLOY_USER` | `root` |
| `HOMELAB_DEPLOY_SSH_KEY` | SSH private key (ed25519) |
| `HOMELAB_DEPLOY_PATH` | `/opt/submatter` (optional) |

Generate deploy key on CT 116:

```bash
ssh-keygen -t ed25519 -f /root/.ssh/github_deploy -N ""
cat /root/.ssh/github_deploy.pub >> /root/.ssh/authorized_keys
# Paste private key into HOMELAB_DEPLOY_SSH_KEY secret
```

Until secrets are set, the deploy workflow skips gracefully (CI still runs).

### Forgejo webhook (optional second trigger)

If Forgejo is canonical git and you want push deploy without Actions:

1. Install [webhook](https://github.com/adhocore/webhook) on CT 116.
2. Copy `proxmox/ct116/webhook.hooks.json.example` → `/etc/webhook.conf`.
3. Forgejo repo → Webhook → POST push events → `http://192.168.68.116:9000/hooks/submatter-deploy`.

---

## Remove from Dokploy

1. Dokploy UI → Submatter/Echodo compose app → Stop → Delete.
2. CT 100 — remove Traefik routes to Dokploy for `echodo.stillwell.cloud` / collab hostname.
3. Update local `AGENT-DEPLOY.md`: add CT 116 row, note Echodo removed from CT 112.

---

## Phase B — OpenCode (preferred)

```bash
export COMPOSE_PROFILES=opencode
# Set in .env: WORKER_SESSION_TYPE=opencode, OPENCODE_SERVER_PASSWORD, OMNIROUTE_*
bash scripts/deploy-prod.sh build   # first time / worker code change
# or: bash scripts/deploy-prod.sh up
```

| Variable | Notes |
|----------|-------|
| `OPENCODE_SERVER_PASSWORD` | `openssl rand -hex 24` — Traefik still fronts Authentik |
| `WORKER_SESSION_TYPE` | `opencode` |
| `OPENCODE_SERVER_URL` | `http://opencode:4096` (Docker DNS) |
| `OMNIROUTE_BASE_URL` | `https://omniroute.stillwell.cloud/v1` or LAN `http://192.168.68.108:20128/v1` |
| `OMNIROUTE_API_KEY` | OmniRoute bearer key |

Web UI: `https://opencode.stillwell.cloud`. Details: `docker/opencode/README.md`.

### Legacy — OpenHands (optional)

```bash
export COMPOSE_PROFILES=agent-server
bash scripts/deploy-prod.sh build
```

| Variable | Notes |
|----------|-------|
| `LOCAL_BACKEND_API_KEY` | `openssl rand -hex 32` |
| `SESSION_API_KEY` | Same as above (worker) |
| `WORKER_SESSION_TYPE` | `openhands` |

Requires Docker socket on CT 116 (`nesting=1` on LXC). Prefer OpenCode unless debugging the old path.
---

## Troubleshooting

| Symptom | Fix |
|---------|-----|
| Build exit 255 ~60–90s | Ensure `deploy-prod.sh` uses `build-prod.sh` (serialized) |
| OAuth broken | DB must be `echodo`, not legacy `tasks` |
| Deploy workflow skipped | Set `HOMELAB_DEPLOY_*` secrets |
| Collab websocket fails | Traefik route to `:1234`, `wss://` in public env vars |
| OpenHands unavailable | `SESSION_API_KEY` must match `LOCAL_BACKEND_API_KEY` |

Local build verify: `pnpm docker:build:prod`

---

## Future: registry pull deploy (no build on CT 116)

Build images on CT 111 CI, push to Forgejo container registry, then on CT 116:

```bash
bash scripts/deploy-prod.sh pull
```

Requires compose `image:` tags instead of `build:` — not implemented yet; track as follow-on if CT 116 builds remain too slow.

---

## Related

- `docs/AGENT-AUTONOMOUS-CONVOYS.md` — convoy + OpenHands roadmap
- `docs/omniroute-setup.md` — OmniRoute operator guide
- `.env.example` — full variable list