224 lines
12 KiB
Markdown
224 lines
12 KiB
Markdown
# Echodo / Submatter — Production Deployment
|
||
|
||
Production runs on **CT 116** (`submatter`, `192.168.68.116`) as plain Docker Compose — not Dokploy.
|
||
|
||
| Doc | Purpose |
|
||
|-----|---------|
|
||
| This file | Operator checklist, env vars, troubleshooting |
|
||
| [`docs/forgejo-migration.md`](forgejo-migration.md) | GitHub → Forgejo + CT 116 cutover |
|
||
| [`proxmox/ct116/README.md`](../proxmox/ct116/README.md) | CT creation, Traefik, auto-deploy, Dokploy removal |
|
||
| `AGENT-DEPLOY.md` (local, gitignored) | Homelab credentials + CT map — add CT 116 row after cutover |
|
||
|
||
---
|
||
|
||
## Why a dedicated CT (not Dokploy)?
|
||
|
||
Echodo builds **five Docker images** from one monorepo; the Next.js compile alone reserves a 5 GB V8 heap. On Dokploy (CT 112), that competes with Zest, Umami, Comms, etc. and OOMs with exit 255.
|
||
|
||
**CT 116** gets the full 8 GB for serialized builds. CT 107 is not an alternative — it still hosts Coolify, Echo Board, Resonance, and other dev apps.
|
||
|
||
| Resource | CT 116 spec |
|
||
|----------|-------------|
|
||
| RAM | 8 GB |
|
||
| CPU | 2 cores |
|
||
| Disk | 32 GB rootfs |
|
||
| Runtime (Phase A) | ~3–4 GB |
|
||
| Phase B (+ OpenHands) | +2 GB |
|
||
|
||
External: Postgres/Redis on **CT 102**, TLS on **CT 100**, OmniRoute on **CT 108**.
|
||
|
||
---
|
||
|
||
## Stack files
|
||
|
||
| File | Use |
|
||
|------|-----|
|
||
| `docker/docker-compose.prod.yml` | Core stack |
|
||
| `docker/docker-compose.prod.standalone.yml` | Publish `:3000` / `:1234` / `:1235` for CT 100 Traefik |
|
||
| `scripts/build-prod.sh` | Serialized image build |
|
||
| `scripts/deploy-prod.sh` | Build + up + health wait |
|
||
|
||
```bash
|
||
docker compose -f docker/docker-compose.prod.yml \
|
||
-f docker/docker-compose.prod.standalone.yml up -d
|
||
```
|
||
|
||
---
|
||
|
||
## Phase A — First deploy on CT 116
|
||
|
||
1. **Create CT 116** — see [`proxmox/ct116/README.md`](../proxmox/ct116/README.md) (`pct create` one-liner).
|
||
2. **Bootstrap** — `bootstrap.sh` installs Docker, clones to `/opt/submatter`.
|
||
3. **Configure `.env`** — copy from `.env.example`; use `echodo` database on CT 102, Redis `/11`.
|
||
4. **Deploy** — `bash scripts/deploy-prod.sh build`
|
||
5. **Traefik** — apply `proxmox/ct116/traefik-ct100.example.yml` on CT 100; remove old Dokploy/Coolify Echodo routes.
|
||
6. **Verify** — `https://echodo.stillwell.cloud`, `wss://collab.echodo.stillwell.cloud`
|
||
|
||
### Required env vars
|
||
|
||
| Variable | Production example | Notes |
|
||
|----------|-------------------|-------|
|
||
| `DATABASE_URL` | `postgresql://echodo:***@192.168.68.102:5432/echodo` | |
|
||
| `REDIS_URL` | `redis://:***@192.168.68.102:6379/11` | |
|
||
| `NEXTAUTH_URL` | `https://echodo.stillwell.cloud` | |
|
||
| `NEXTAUTH_SECRET` / `AUTH_SECRET` | `openssl rand -base64 32` | |
|
||
| `AUTH_AUTHENTIK_*` | | Authentik OIDC app `echodo` |
|
||
| `COLLAB_SERVER_URL` | `wss://collab.echodo.stillwell.cloud` | Server-side |
|
||
| `NEXT_PUBLIC_COLLAB_SERVER_URL` | `wss://collab.echodo.stillwell.cloud` | **Build arg** — requires image rebuild |
|
||
| `NEXT_PUBLIC_CHAT_SERVER_URL` | `wss://echodo-chat.stillwell.cloud` | **Build arg** — requires web image rebuild; provision DNS/TLS first |
|
||
| `CHAT_TICKET_SECRET` | Generate with `openssl rand -base64 32` | Same runtime secret for web and collab; at least 32 characters; never a build arg |
|
||
| `CHAT_PORT` | `1235` | Second listener inside collab; cannot equal Hocuspocus port `1234` |
|
||
| `OMNIROUTE_BASE_URL` | `https://omniroute.stillwell.cloud/v1` | Web + worker |
|
||
| `OMNIROUTE_API_KEY` | (your key) | + register in Settings → AI → Providers |
|
||
| `FORGEJO_TOKEN` / `GITHUB_TOKEN` | PAT values | Referenced by name in Connected repos settings |
|
||
| `NEXT_PUBLIC_UMAMI_*` | | Analytics (CT 112 Umami) |
|
||
|
||
Changing any `NEXT_PUBLIC_*` value requires `bash scripts/deploy-prod.sh build`, not just `docker compose restart`.
|
||
|
||
---
|
||
|
||
## Workspace Chat integration gate
|
||
|
||
Chat runs inside the existing `collab` service on a **separate listener** from Hocuspocus. The worker publishes metadata-only Redis hints; Postgres history and mutation catch-up remain authoritative. Team Chat does not invoke AI agents, and object contexts are not enabled by this deployment.
|
||
|
||
1. Set the Chat variables above in `/opt/submatter/.env`. Keep the signing secret server-side; Compose requires a nonempty value, and the application requires at least 32 characters. A missing/short secret disables the listener and makes the dual-listener health check fail.
|
||
2. Provision `echodo-chat.stillwell.cloud` DNS and TLS on CT 100 (canonical browser endpoint). `chat.stillwell.cloud` belongs to another app: never include it in the Workspace Chat router. Apply the `echodo-chat` router/service from [`traefik-ct100.example.yml`](../proxmox/ct116/traefik-ct100.example.yml). Traefik supports WebSocket upgrades without a separate service process. If `CHAT_PORT` changes, update the proxy backend port too. Restrict host backend ports to the trusted proxy network.
|
||
|
||
For the hostname migration, set `NEXT_PUBLIC_CHAT_SERVER_URL=wss://echodo-chat.stillwell.cloud` in CT 116's `/opt/submatter/.env`, then run from `/opt/submatter`:
|
||
|
||
```bash
|
||
docker compose --project-directory /opt/submatter -f docker/docker-compose.prod.yml -f docker/docker-compose.prod.standalone.yml build web
|
||
docker compose --project-directory /opt/submatter -f docker/docker-compose.prod.yml -f docker/docker-compose.prod.standalone.yml up -d --no-deps web
|
||
```
|
||
|
||
A git push or container restart alone does not change the baked browser URL. Test Workspace Chat inside `https://echodo.stillwell.cloud`; its WebSocket should connect to `wss://echodo-chat.stillwell.cloud/chat`. The WSS endpoint is not a separate chat UI.
|
||
3. Before rolling out, verify the database target is CT 102 **`echodo`** and inspect `drizzle.__drizzle_migrations`. The operator reports 24 ledger entries through `0023`, verified Chat schema, and ownership of all `chat_*` tables transferred to `echodo`. Do not reapply these migrations; inspect the ledger for any future pending entries. Compare the ledger with `packages/database/migrations/meta/_journal.json`; apply only pending journaled migrations using `pnpm --filter @tasks/database db:migrate` in a checkout with the real `DATABASE_URL` exported securely. Do not print the URL, use `db:push`, or replay historical migrations manually. Take a backup first.
|
||
4. Verify `0022` added `chat_messages.send_intent_fingerprint` and positive `revision`; verify `0023` added `chat_message_mutations`, its tenant/channel/message foreign keys and indexes, and `chat_channels.next_mutation_sequence`. Do not infer applied schema from TypeScript files. The historical `0017` correction is already local; do not replay it on an initialized database.
|
||
5. Validate Compose on the deployment host without printing resolved secrets:
|
||
|
||
```bash
|
||
docker compose --project-directory /opt/submatter --env-file /opt/submatter/.env \
|
||
-f docker/docker-compose.prod.yml -f docker/docker-compose.prod.standalone.yml config --quiet
|
||
bash scripts/deploy-prod.sh build
|
||
```
|
||
|
||
6. Confirm both collab listeners and worker health. The TCP health check verifies listener liveness only, **not** Redis delivery, database authorization, or end-to-end readiness. An ordinary HTTP request to the Chat listener returns 404; use an authenticated browser WebSocket session for acceptance.
|
||
7. Test two signed-in workspace members: send/reply/edit/delete, unread acknowledgement, reconnect catch-up, offline edits/deletes, and duplicate retries. Revoke private/direct membership while connected and verify subsequent hints/history are denied and cached content is cleared; admins must not bypass membership. Verify another workspace cannot obtain history or subscribe. Restart collab/worker and test Redis outage recovery without message loss.
|
||
|
||
Run PostgreSQL integration tests only against a disposable test database, not production. DNS, TLS, migrations, Redis/WebSocket delivery, and two-browser acceptance are independent release gates; passing unit tests does not satisfy them. See [Current Progress](Current%20Progress.md) for the current evidence and blockers.
|
||
|
||
### Post-go-live reconciliation
|
||
|
||
The operator reports Chat live with 14/14 acceptance checks; see [Current Progress](Current%20Progress.md) for provenance and scope. Rebuild web on the next deploy to replace hot-patched image chunks with source-built code. The reconciled ticket route uses `AUTH_URL` or `NEXTAUTH_URL` as its trusted public origin; set it to the HTTPS **web app** origin, not the WSS endpoint. It ignores forwarded headers for this decision and fails closed without valid production configuration. Restrict direct backend access to the proxy network regardless.
|
||
|
||
`AUTH_DEV_PASSWORD` was enabled for acceptance. This shared password can authenticate existing email identities; it is not individual-user password authentication. Confirm SSO works, then remove it and recreate web if it was temporary. Do not leave this enabled solely because acceptance needed it. Re-run ticket origin/authentication and live Chat acceptance after the source-built deployment.
|
||
|
||
## Auto-deploy on push
|
||
|
||
**CI** (every PR): `.github/workflows/ci.yml` — lint, type-check, test on GitHub-hosted or PR runner.
|
||
|
||
**CD** (merge to `main`): `.github/workflows/deploy-homelab.yml` and `.forgejo/workflows/deploy-homelab.yml` — SSH from CT 111 `axiom` runner → CT 116 → `deploy-prod.sh build`.
|
||
|
||
### GitHub / Forgejo repo secrets
|
||
|
||
| Secret | Value |
|
||
|--------|-------|
|
||
| `HOMELAB_DEPLOY_HOST` | `192.168.68.116` |
|
||
| `HOMELAB_DEPLOY_USER` | `root` |
|
||
| `HOMELAB_DEPLOY_SSH_KEY` | SSH private key (ed25519) |
|
||
| `HOMELAB_DEPLOY_PATH` | `/opt/submatter` (optional) |
|
||
|
||
Generate deploy key on CT 116:
|
||
|
||
```bash
|
||
ssh-keygen -t ed25519 -f /root/.ssh/github_deploy -N ""
|
||
cat /root/.ssh/github_deploy.pub >> /root/.ssh/authorized_keys
|
||
# Paste private key into HOMELAB_DEPLOY_SSH_KEY secret
|
||
```
|
||
|
||
Until secrets are set, the deploy workflow skips gracefully (CI still runs).
|
||
|
||
### Forgejo webhook (optional second trigger)
|
||
|
||
If Forgejo is canonical git and you want push deploy without Actions:
|
||
|
||
1. Install [webhook](https://github.com/adhocore/webhook) on CT 116.
|
||
2. Copy `proxmox/ct116/webhook.hooks.json.example` → `/etc/webhook.conf`.
|
||
3. Forgejo repo → Webhook → POST push events → `http://192.168.68.116:9000/hooks/submatter-deploy`.
|
||
|
||
---
|
||
|
||
## Remove from Dokploy
|
||
|
||
1. Dokploy UI → Submatter/Echodo compose app → Stop → Delete.
|
||
2. CT 100 — remove Traefik routes to Dokploy for `echodo.stillwell.cloud` / collab hostname.
|
||
3. Update local `AGENT-DEPLOY.md`: add CT 116 row, note Echodo removed from CT 112.
|
||
|
||
---
|
||
|
||
## Phase B — OpenCode (preferred)
|
||
|
||
```bash
|
||
export COMPOSE_PROFILES=opencode
|
||
# Set in .env: WORKER_SESSION_TYPE=opencode, OPENCODE_SERVER_PASSWORD, OMNIROUTE_*
|
||
bash scripts/deploy-prod.sh build # first time / worker code change
|
||
# or: bash scripts/deploy-prod.sh up
|
||
```
|
||
|
||
| Variable | Notes |
|
||
|----------|-------|
|
||
| `OPENCODE_SERVER_PASSWORD` | `openssl rand -hex 24` — Traefik still fronts Authentik |
|
||
| `WORKER_SESSION_TYPE` | `opencode` |
|
||
| `OPENCODE_SERVER_URL` | `http://opencode:4096` (Docker DNS) |
|
||
| `OMNIROUTE_BASE_URL` | `https://omniroute.stillwell.cloud/v1` or LAN `http://192.168.68.108:20128/v1` |
|
||
| `OMNIROUTE_API_KEY` | OmniRoute bearer key |
|
||
|
||
Web UI: `https://opencode.stillwell.cloud`. Details: `docker/opencode/README.md`.
|
||
|
||
### Legacy — OpenHands (optional)
|
||
|
||
```bash
|
||
export COMPOSE_PROFILES=agent-server
|
||
bash scripts/deploy-prod.sh build
|
||
```
|
||
|
||
| Variable | Notes |
|
||
|----------|-------|
|
||
| `LOCAL_BACKEND_API_KEY` | `openssl rand -hex 32` |
|
||
| `SESSION_API_KEY` | Same as above (worker) |
|
||
| `WORKER_SESSION_TYPE` | `openhands` |
|
||
|
||
Requires Docker socket on CT 116 (`nesting=1` on LXC). Prefer OpenCode unless debugging the old path.
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Fix |
|
||
|---------|-----|
|
||
| Build exit 255 ~60–90s | Ensure `deploy-prod.sh` uses `build-prod.sh` (serialized) |
|
||
| OAuth broken | DB must be `echodo`, not legacy `tasks` |
|
||
| Deploy workflow skipped | Set `HOMELAB_DEPLOY_*` secrets |
|
||
| Collab websocket fails | Traefik route to `:1234`, `wss://` in public env vars |
|
||
| OpenHands unavailable | `SESSION_API_KEY` must match `LOCAL_BACKEND_API_KEY` |
|
||
|
||
Local build verify: `pnpm docker:build:prod`
|
||
|
||
---
|
||
|
||
## Future: registry pull deploy (no build on CT 116)
|
||
|
||
Build images on CT 111 CI, push to Forgejo container registry, then on CT 116:
|
||
|
||
```bash
|
||
bash scripts/deploy-prod.sh pull
|
||
```
|
||
|
||
Requires compose `image:` tags instead of `build:` — not implemented yet; track as follow-on if CT 116 builds remain too slow.
|
||
|
||
---
|
||
|
||
## Related
|
||
|
||
- `docs/AGENT-AUTONOMOUS-CONVOYS.md` — convoy + OpenHands roadmap
|
||
- `docs/omniroute-setup.md` — OmniRoute operator guide
|
||
- `.env.example` — full variable list
|