Submatter/docs/adr-model-routing.md

# ADR: Model routing and provider registry

**Status:** Accepted (draft implementation)  
**Date:** 2026-07-03  
**Context:** Echodo automation foundation — orchestrator, worker, and mission-control UI without Cursor as the runtime substrate.

## Decision

Echodo will own a **workspace-scoped model provider registry** and an **extensible role → model routing table**. Every automated pipeline run resolves a concrete provider configuration before inference. Resolution is deterministic, logged on every run, and configurable from Settings without redeploying orchestrator or worker code.

## Problem

The agent-pipeline defines per-role models via:

- `model:` frontmatter in `.cursor/agents/role-*.md` (Cursor-only enforcement)
- `model_policy.roles` in convoy frontmatter (recommendation only)
- `recommended_model` on implementer briefs (optional override)

That works for manual Cursor sessions. It does **not** work when:

- An orchestrator dispatches `role-implementer` unattended
- Planning uses Claude API and implementation uses local Ollama on a Mac
- Operators add providers (OpenAI, Gemini, NVIDIA NIM, LiteLLM gateway) without code changes
- Custom roles (e.g. `gate-copilot`, `role-security-auditor`) need routing

We need one contract that web chat, orchestrator, worker, and MCP logging all share.

## Goals

1. **Per-role model selection** — conductor/architect on cloud; implementer/auditors on local coder models.
2. **Extensible providers** — Anthropic, OpenAI, Google (Gemini), OpenAI-compatible (Ollama, LiteLLM, NVIDIA NIM, Azure OpenAI).
3. **Extensible roles** — seed pipeline’s nine L2 roles; allow arbitrary `roleId` rows.
4. **Layered overrides** — workspace defaults → convoy `model_policy` → brief `recommended_model`.
5. **No secrets in DB** — store `apiKeyEnvVar` names; values live in deploy env (Coolify/Dokploy/Mac).
6. **Observable** — every `agent_runs` / `convoy_events` row records alias, provider type, model id, tier.

## Non-goals (this ADR)

- Implementing the orchestrator or worker (separate epics).
- LiteLLM as a required dependency (optional alias pointing at a gateway URL).
- Storing API keys in Postgres or committed YAML.
- Replacing `docs/model-routing-policy.md` tier guidance — this ADR **implements** it in Echodo.

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│  Settings → AI → Providers     (ModelProviderAlias CRUD)    │
│  Settings → AI → Role routing    (RoleModelRoute CRUD)        │
└────────────────────────────┬────────────────────────────────┘
                             │
                    resolveModel(workspaceId, roleId, ctx?)
                             │
         ┌───────────────────┼───────────────────┐
         ▼                   ▼                   ▼
   apps/web            apps/orchestrator    apps/worker
   (chat, gate          (dispatch job        (inference
    co-pilot)            with ModelRef)       only)
```

### Resolution order (first match wins)

1. **Brief override** — `recommended_model` alias on `.convoys/<slug>/brief-N-*.md`
2. **Convoy override** — `model_policy.roles[roleId]` in `.convoys/<slug>.md`
3. **Workspace route** — `role_model_routes` row for `(workspace_id, role_id)`
4. **Pipeline template** — shipped defaults (nine L2 roles + `gate-copilot`, `chat-default`)

If the alias is missing or disabled, fail closed with a structured error — do not silently fall back to a hardcoded model.

### Escalation

When `model_policy.escalate_to` is set (convoy) or workspace policy defines escalation:

- After N failed implementer attempts (default: 2), orchestrator re-dispatches the same brief with the escalation alias.
- Log `model_tier` change on the new run.

## Data model

### `model_providers` (per workspace)

| Column | Type | Notes |
|--------|------|-------|
| `id` | uuid PK | |
| `workspace_id` | uuid FK | Required; all queries filter by this |
| `alias` | varchar | Unique per workspace, e.g. `claude-planning` |
| `display_name` | varchar | UI label |
| `provider_type` | enum | See below |
| `model_id` | varchar | Provider-native model string |
| `base_url` | text nullable | Required for `openai_compatible` |
| `api_key_env_var` | varchar nullable | e.g. `ANTHROPIC_API_KEY` |
| `extra_headers` | jsonb nullable | Optional HTTP headers |
| `default_params` | jsonb nullable | `temperature`, `maxTokens`, … |
| `metadata` | jsonb nullable | `tier`, cost hints for dashboard |
| `enabled` | boolean | Default true |
| `created_at` / `updated_at` | timestamptz | |

**`provider_type` enum:**

| Value | Client | Examples |
|-------|--------|----------|
| `anthropic` | `@ai-sdk/anthropic` | Claude API |
| `openai` | `@ai-sdk/openai` | OpenAI API |
| `google` | `@ai-sdk/google` | Gemini |
| `openai_compatible` | `@ai-sdk/openai` + `baseURL` | Ollama, LiteLLM, **NVIDIA NIM**, vLLM |
| `azure_openai` | `@ai-sdk/openai` + Azure config | Azure deployments |

Future: `custom` with a registered adapter — not in v1.

### `role_model_routes` (per workspace)

| Column | Type | Notes |
|--------|------|-------|
| `id` | uuid PK | |
| `workspace_id` | uuid FK | |
| `role_id` | varchar | e.g. `role-implementer`, `gate-copilot` |
| `provider_alias_id` | uuid FK → `model_providers.id` | |
| `enabled` | boolean | |
| `sort_order` | int | UI ordering |
| `created_at` / `updated_at` | timestamptz | |

Unique constraint: `(workspace_id, role_id)`.

`role_id` is a **string**, not a DB enum — new pipeline roles do not require migrations.

### Resolved config (runtime, not persisted)

```typescript
type ResolvedModel = {
  alias: string;
  providerType: ProviderType;
  modelId: string;
  baseUrl?: string;
  apiKey: string; // read from process.env[apiKeyEnvVar] at call time
  extraHeaders?: Record<string, string>;
  defaultParams?: { temperature?: number; maxTokens?: number };
  tier?: "premium" | "fast" | "local" | "auto";
};
```

Pure function: `resolveModel(input, db)` lives in `packages/shared` (types + zod) with implementation in `packages/ai` or `packages/shared` (no React, no DB imports in shared — DB fetch in tRPC/orchestrator, pass rows into resolver).

## Provider examples

### Local Ollama (Mac)

```yaml
alias: qwen-coder
provider_type: openai_compatible
model_id: qwen2.5-coder:32b
base_url: http://192.168.1.42:11434/v1
api_key_env_var: OLLAMA_API_KEY  # any non-empty string Ollama accepts
metadata: { tier: local }
```

### Local MLX on Apple Silicon (Mac)

[MLX](https://github.com/ml-explore/mlx) is Apple's on-device ML framework. Echodo does **not** talk to MLX directly — use an **OpenAI-compatible HTTP server** in front of MLX, same `provider_type: openai_compatible` as Ollama.

| Server | Package | Default port | When to use |
|--------|---------|--------------|-------------|
| **mlx_lm.server** | `pip install mlx-lm` | 8080 | Solo dev, single implementer, simplest path |
| **mlx-openai-server** | [cubist38/mlx-openai-server](https://github.com/cubist38/mlx-openai-server) | 8000 | **Parallel implementer fan-out** — continuous batching handles concurrent requests better than the reference server |
| mlx-lm-server (community) | [0xDaizz/mlx-lm-server](https://github.com/0xDaizz/mlx-lm-server) | 8000 | Alternative if mlx-openai-server doesn't fit |

**Parallel implementers note:** When the orchestrator dispatches multiple `role-implementer` jobs at once (disjoint briefs per agent-pipeline `slice_dependencies`), a single local inference process can become the bottleneck. Prefer **mlx-openai-server** (or mlx-lm-server with batching) over the minimal `mlx_lm.server` when running 2+ implementers against the same Mac. Ollama can also serve parallel requests but MLX-native models (`mlx-community/*` on Hugging Face) often perform better on Apple Silicon unified memory. You may run **both** Ollama (`:11434`) and an MLX server (`:8000` / `:8080`) as separate registry aliases and route roles accordingly.

```yaml
alias: qwen-coder-mlx
provider_type: openai_compatible
model_id: mlx-community/Qwen2.5-Coder-32B-Instruct-4bit
base_url: http://192.168.1.42:8000/v1   # mlx-openai-server; use :8080/v1 for mlx_lm.server
api_key_env_var: MLX_API_KEY            # placeholder; local servers ignore it
metadata: { tier: local, runtime: mlx }
```

**Operator setup (mlx-openai-server example):**

```bash
# Machine-wide install (all repos): ~/mlx-inference/mlx-serve.sh
# See ~/mlx-inference/README.md
~/mlx-inference/mlx-serve.sh

# Or manual:
pip install mlx-lm mlx-openai-server
mlx-openai-server launch \
  --model-path mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
  --model-type lm \
  --host 0.0.0.0 \
  --port 8000
```

**Networking:** Bind `0.0.0.0` when Echodo (Coolify) or `apps/worker` on another host must reach the Mac over LAN. Restrict firewall to homelab subnet. Do not expose raw local inference to the public internet.

**v1 registry:** No `provider_type: mlx` — HTTP gateway only. Native MLX SDK in `apps/worker` is out of scope.


```yaml
alias: claude-planning
provider_type: anthropic
model_id: claude-sonnet-4-20250514
api_key_env_var: ANTHROPIC_API_KEY
metadata: { tier: premium }
```

### NVIDIA NIM

```yaml
alias: nim-llama-70b
provider_type: openai_compatible
model_id: meta/llama-3.1-70b-instruct
base_url: https://integrate.api.nvidia.com/v1
api_key_env_var: NVIDIA_API_KEY
```

### Gemini

```yaml
alias: gemini-flash
provider_type: google
model_id: gemini-2.0-flash
api_key_env_var: GOOGLE_GENERATIVE_AI_API_KEY
```

### LiteLLM gateway (optional)

```yaml
alias: litellm-gateway
provider_type: openai_compatible
model_id: qwen2.5-coder:32b  # or per-role model_id in route table
base_url: http://host:4000/v1
api_key_env_var: LITELLM_MASTER_KEY
```

## Default role routing (seed template)

Shipped when a workspace enables pipeline integration:

| role_id | Suggested alias | tier |
|---------|-----------------|------|
| `role-conductor` | `claude-planning` | premium |
| `role-architect` | `claude-planning` | premium |
| `role-ia-architect` | `claude-planning` | premium |
| `role-ux-reviewer` | `qwen-coder` or `qwen-coder-mlx` | fast/local |
| `role-implementer` | `qwen-coder` or `qwen-coder-mlx` | fast/local |
| `role-reviewer` | `qwen-coder` or `qwen-coder-mlx` | fast/local |
| `role-design-system-auditor` | `qwen-coder` or `qwen-coder-mlx` | fast/local |
| `role-a11y-auditor` | `qwen-coder` or `qwen-coder-mlx` | fast/local |
| `role-doc-writer` | `qwen-coder` or `qwen-coder-mlx` | fast/local |
| `gate-copilot` | `claude-planning` | premium |
| `chat-default` | workspace choice | — |

Operators remap aliases in Settings; seed is not immutable.

## Convoy / brief integration

Convoy frontmatter uses **registry aliases**, not Cursor model slugs:

```yaml
model_policy:
  roles:
    role-conductor: claude-planning
    role-architect: claude-planning
    role-implementer: qwen-coder
  escalate_to: claude-planning
```

Brief frontmatter:

```yaml
recommended_model: qwen-coder
model_tier: fast
```

Importer/parser for convoy files is out of scope for the first epic; orchestrator reads frontmatter when dispatching.

## API surface (tRPC)

Router: `ai.models` (or `pipeline.models`)

| Procedure | Purpose |
|-----------|---------|
| `listProviders` | Paginated; workspace-scoped |
| `createProvider` / `updateProvider` / `deleteProvider` | Admin-only |
| `testProvider` | One-token completion; returns latency + error detail |
| `listRoleRoutes` | All routes for workspace |
| `upsertRoleRoute` | Set role → alias; supports custom `roleId` |
| `deleteRoleRoute` | Remove custom role |
| `resolveForRole` | Debug: given `roleId` + optional convoy/brief context |

## UI

- `/{workspaceSlug}/settings/ai/providers` — provider table + add/edit dialog + test button
- `/{workspaceSlug}/settings/ai/role-routing` — role → alias table + add custom role
- Link from convoy/gate screens: “Model for this step: `qwen-coder` (change)”

## Logging contract

Extend `agent_runs` (and future `convoy_events`) with:

- `model_alias` varchar nullable
- `provider_type` varchar nullable
- `model_id` varchar nullable
- `model_tier` varchar nullable

`complete_task` MCP tool accepts optional `modelAlias` / `modelTier` for close-out logging.

## Security

- Provider CRUD: workspace admin only (same bar as `updateWorkflowPrompt`).
- `testProvider` runs server-side; rate-limit per workspace.
- Never return resolved API keys to the browser — only alias and env var **name**.
- Worker/orchestrator processes must have env vars for aliases they use.

## Relationship to other ADRs / plans

| Artifact | Relationship |
|----------|----------------|
| `docs/model-routing-policy.md` | Tier guidance; Cursor-era slugs migrate to aliases |
| `plans/Plan-agent-coordination` | `claim_task` / `complete_task` log model fields |
| `plans/Plan-automation-foundation` | Implements this ADR (model registry epic) |
| Future orchestrator epic | Calls `resolveModel` on every dispatch |
| Future worker epic | Consumes `ResolvedModel` only |

## Migration from Cursor slugs

Legacy convoys may reference `claude-4.6-opus-high-thinking` or `composer-2.5-fast`. Optional alias map in workspace metadata:

```json
{
  "legacy_model_aliases": {
    "claude-4.6-opus-high-thinking": "claude-planning",
    "composer-2.5-fast": "qwen-coder"
  }
}
```

## Consequences

**Positive**

- Single place to change Claude vs Qwen vs NIM without code deploys.
- Orchestrator and worker stay provider-agnostic.
- Extensible roles support pipeline evolution.

**Negative**

- Operators must configure providers before automation runs (mitigated: test button + clear errors).
- Two sources of truth for convoy overrides (markdown + DB) until convoy lifecycle MCP lands — resolution order is explicit.

## Open questions

1. **Repo-level `echodo.models.yml`** — mirror workspace config in git for bootstrap? Deferred; DB is v1 source.
2. **Per-user overrides** — deferred; workspace-level only for v1.
3. **Budget caps per alias** — defer to LiteLLM or Phase 4 workers.

## References

- Agent-pipeline: `docs/model-routing-policy.md`, `docs/role-reference.md`
- Agent-pipeline v0.4 plan: convoy lifecycle + Echodo integration
- Vercel AI SDK provider packages in `packages/ai`
- MLX: [ml-explore/mlx](https://github.com/ml-explore/mlx), [mlx-lm SERVER.md](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md), [mlx-openai-server](https://github.com/cubist38/mlx-openai-server)
- Free-tier discovery: [cheahjs/free-llm-api-resources](https://github.com/cheahjs/free-llm-api-resources), [mnfst/awesome-free-llm-apis](https://github.com/mnfst/awesome-free-llm-apis)