Submatter/proxmox/ct116/README.md

# CT 116 — Submatter / Echodo (production)

Dedicated LXC for the multitenant task app + convoy worker stack. Replaces Dokploy (CT 112) and eventually decommissions the Echodo slice on CT 107 Coolify.

## Spec (from AGENT-DEPLOY.md sizing)

| Setting | Value | Why |
|---------|-------|-----|
| **CT ID** | **116** | Next free slot (114 planned Nextcloud, 115 Forgejo) |
| **Hostname** | `submatter` | Product name; DNS stays `echodo.stillwell.cloud` |
| **IP** | `192.168.68.116/22` | Sequential homelab LAN |
| **RAM** | **8192 MB** | Web build needs ~5 GB V8 heap + BuildKit headroom |
| **CPU** | 2 cores | Same as former CT 107 Echodo deploy |
| **Disk** | **32 GB** | Five images + layer cache (AGENT-DEPLOY: 30–32 GB for compose-heavy CTs) |
| **Features** | `nesting=1` | Required for Docker-in-Docker (Phase B OpenHands sandboxes) |

**Do not reuse CT 107** for this — it still runs Coolify, Echo Board, Resonance, pipeline dashboard, Amazon scraper, etc. Competing for RAM during builds was the original OOM problem.

## External services (unchanged)

| Service | Host |
|---------|------|
| Postgres `echodo` DB | CT 102 `:5432` |
| Redis DB `/11` | CT 102 `:6379` |
| TLS / public routing | CT 100 Traefik |
| OmniRoute (Phase B) | CT 108 |
| Umami analytics | CT 112 (script URL only) |

## Clone URL

Production checkout: **https://forgejo.stillwell.cloud/rstillwell/Submatter.git** (`main`).

Migration from GitHub: [`docs/forgejo-migration.md`](../../docs/forgejo-migration.md).

## Bootstrap

On Proxmox host:

```bash
pct create 116 local:vztmpl/debian-13-standard_13.0-1_amd64.tar.zst \
  --hostname submatter --cores 2 --memory 8192 --swap 512 \
  --rootfs local-lvm:32 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.68.116/22,gw=192.168.68.1 \
  --unprivileged 1 --features nesting=1,keyctl=1

pct start 116
./proxmox/scripts/sync.sh push 116   # when homelab sync script includes this tree
pct exec 116 -- bash /root/bootstrap.sh
```

Edit `/opt/submatter/.env`, then:

```bash
pct exec 116 -- bash -lc 'cd /opt/submatter && bash scripts/deploy-prod.sh build'
```

## Routing

1. Merge `traefik-ct100.example.yml` into CT 100 dynamic config.
2. Remove old Echodo routes pointing at CT 107 Coolify or CT 112 Dokploy once cutover is verified.
3. AdGuard wildcard `*.stillwell.cloud` → CT 100 (unchanged).

## Delete from Dokploy (CT 112)

In Dokploy UI:

1. Open the Submatter/Echodo compose application.
2. Stop containers → Delete application (keep CT 112 shared services like Umami).
3. Remove any CT 100 Traefik routes that pointed at Dokploy for `echodo.stillwell.cloud`.

## Auto-deploy on push

Three options (can combine GitHub + Forgejo):

### A. Self-hosted runner (recommended)

Uses CT 111 `axiom` runners (already org-scoped to `stwl-labs`).

Repo secrets:

| Secret | Value |
|--------|-------|
| `HOMELAB_DEPLOY_HOST` | `192.168.68.116` |
| `HOMELAB_DEPLOY_USER` | `root` (or dedicated deploy user) |
| `HOMELAB_DEPLOY_SSH_KEY` | private key allowed to SSH to CT 116 |
| `HOMELAB_DEPLOY_PATH` | `/opt/submatter` (optional) |

Workflows:

- `.github/workflows/deploy-homelab.yml` — GitHub pushes to `main`
- `.forgejo/workflows/deploy-homelab.yml` — Forgejo Actions (same file)

CI (lint/type-check/test) still runs on every PR via `.github/workflows/ci.yml`. Deploy runs **after** merge to `main`.

### B. Forgejo webhook (no Actions)

On Forgejo → repo → Webhooks:

- URL: `http://192.168.68.116:9000/hooks/deploy` (front with Caddy/`webhook` tool)
- Secret: set `DEPLOY_WEBHOOK_SECRET` on CT 116
- Events: Push

Handler: `deploy-webhook.sh`

### C. Manual

```bash
ssh root@192.168.68.116 'cd /opt/submatter && git pull && bash scripts/deploy-prod.sh build'
```

## Phase B (OpenCode — preferred)

```bash
export COMPOSE_PROFILES=opencode
bash scripts/deploy-prod.sh build
```

Set in `.env`: `WORKER_SESSION_TYPE=opencode`, `OPENCODE_SERVER_PASSWORD`, `OMNIROUTE_API_KEY` / `OMNIROUTE_BASE_URL`. See `docker/opencode/README.md`.

### Legacy Phase B (OpenHands)

```bash
export COMPOSE_PROFILES=agent-server
bash scripts/deploy-prod.sh build
```

Set `WORKER_SESSION_TYPE=openhands`, `SESSION_API_KEY`, `LOCAL_BACKEND_API_KEY`, OmniRoute vars in `.env`.
## AGENT-DEPLOY.md handoff

After CT is live, add to your local `AGENT-DEPLOY.md` (gitignored):

```markdown
| 116 | LXC | submatter | 192.168.68.116 | 8 GB | 2 | 32 GB | ACTIVE | Submatter/Echodo production (docker compose) |
```

Move Echodo row off CT 107 / CT 112 sections.

## Related

- [`docs/deployment-production.md`](../../docs/deployment-production.md) — env vars, troubleshooting
- [`docker/docker-compose.prod.yml`](../../docker/docker-compose.prod.yml) — stack definition