Submatter/docs/adr-model-routing.md

ADR: Model routing and provider registry

Status: Accepted (draft implementation)
Date: 2026-07-03
Context: Echodo automation foundation — orchestrator, worker, and mission-control UI without Cursor as the runtime substrate.

Decision

Echodo will own a workspace-scoped model provider registry and an extensible role → model routing table. Every automated pipeline run resolves a concrete provider configuration before inference. Resolution is deterministic, logged on every run, and configurable from Settings without redeploying orchestrator or worker code.

Problem

The agent-pipeline defines per-role models via:

• model: frontmatter in .cursor/agents/role-*.md (Cursor-only enforcement)
• model_policy.roles in convoy frontmatter (recommendation only)
• recommended_model on implementer briefs (optional override)

That works for manual Cursor sessions. It does not work when:

• An orchestrator dispatches role-implementer unattended
• Planning uses Claude API and implementation uses local Ollama on a Mac
• Operators add providers (OpenAI, Gemini, NVIDIA NIM, LiteLLM gateway) without code changes
• Custom roles (e.g. gate-copilot, role-security-auditor) need routing

We need one contract that web chat, orchestrator, worker, and MCP logging all share.

Goals

1. Per-role model selection — conductor/architect on cloud; implementer/auditors on local coder models.
2. Extensible providers — Anthropic, OpenAI, Google (Gemini), OpenAI-compatible (Ollama, LiteLLM, NVIDIA NIM, Azure OpenAI).
3. Extensible roles — seed pipeline’s nine L2 roles; allow arbitrary roleId rows.
4. Layered overrides — workspace defaults → convoy model_policy → brief recommended_model.
5. No secrets in DB — store apiKeyEnvVar names; values live in deploy env (Coolify/Dokploy/Mac).
6. Observable — every agent_runs / convoy_events row records alias, provider type, model id, tier.

Non-goals (this ADR)

• Implementing the orchestrator or worker (separate epics).
• LiteLLM as a required dependency (optional alias pointing at a gateway URL).
• Storing API keys in Postgres or committed YAML.
• Replacing docs/model-routing-policy.md tier guidance — this ADR implements it in Echodo.

Architecture

┌─────────────────────────────────────────────────────────────┐
│  Settings → AI → Providers     (ModelProviderAlias CRUD)    │
│  Settings → AI → Role routing    (RoleModelRoute CRUD)        │
└────────────────────────────┬────────────────────────────────┘
                             │
                    resolveModel(workspaceId, roleId, ctx?)
                             │
         ┌───────────────────┼───────────────────┐
         ▼                   ▼                   ▼
   apps/web            apps/orchestrator    apps/worker
   (chat, gate          (dispatch job        (inference
    co-pilot)            with ModelRef)       only)

Resolution order (first match wins)

1. Brief override — recommended_model alias on .convoys/<slug>/brief-N-*.md
2. Convoy override — model_policy.roles[roleId] in .convoys/<slug>.md
3. Workspace route — role_model_routes row for (workspace_id, role_id)
4. Pipeline template — shipped defaults (nine L2 roles + gate-copilot, chat-default)

If the alias is missing or disabled, fail closed with a structured error — do not silently fall back to a hardcoded model.

Escalation

When model_policy.escalate_to is set (convoy) or workspace policy defines escalation:

• After N failed implementer attempts (default: 2), orchestrator re-dispatches the same brief with the escalation alias.
• Log model_tier change on the new run.

Data model

model_providers (per workspace)

Column	Type	Notes
id	uuid PK
workspace_id	uuid FK	Required; all queries filter by this
alias	varchar	Unique per workspace, e.g. claude-planning
display_name	varchar	UI label
provider_type	enum	See below
model_id	varchar	Provider-native model string
base_url	text nullable	Required for openai_compatible
api_key_env_var	varchar nullable	e.g. ANTHROPIC_API_KEY
extra_headers	jsonb nullable	Optional HTTP headers
default_params	jsonb nullable	temperature, maxTokens, …
metadata	jsonb nullable	tier, cost hints for dashboard
enabled	boolean	Default true
created_at / updated_at	timestamptz

provider_type enum:

Value	Client	Examples
anthropic	@ai-sdk/anthropic	Claude API
openai	@ai-sdk/openai	OpenAI API
google	@ai-sdk/google	Gemini
openai_compatible	@ai-sdk/openai + baseURL	Ollama, LiteLLM, NVIDIA NIM, vLLM
azure_openai	@ai-sdk/openai + Azure config	Azure deployments

Future: custom with a registered adapter — not in v1.

role_model_routes (per workspace)

Column	Type	Notes
id	uuid PK
workspace_id	uuid FK
role_id	varchar	e.g. role-implementer, gate-copilot
provider_alias_id	uuid FK → model_providers.id
enabled	boolean
sort_order	int	UI ordering
created_at / updated_at	timestamptz

Unique constraint: (workspace_id, role_id).

role_id is a string, not a DB enum — new pipeline roles do not require migrations.

Resolved config (runtime, not persisted)

type ResolvedModel = {
  alias: string;
  providerType: ProviderType;
  modelId: string;
  baseUrl?: string;
  apiKey: string; // read from process.env[apiKeyEnvVar] at call time
  extraHeaders?: Record<string, string>;
  defaultParams?: { temperature?: number; maxTokens?: number };
  tier?: "premium" | "fast" | "local" | "auto";
};

Pure function: resolveModel(input, db) lives in packages/shared (types + zod) with implementation in packages/ai or packages/shared (no React, no DB imports in shared — DB fetch in tRPC/orchestrator, pass rows into resolver).

Provider examples

Local Ollama (Mac)

alias: qwen-coder
provider_type: openai_compatible
model_id: qwen2.5-coder:32b
base_url: http://192.168.1.42:11434/v1
api_key_env_var: OLLAMA_API_KEY  # any non-empty string Ollama accepts
metadata: { tier: local }

Local MLX on Apple Silicon (Mac)

MLX is Apple's on-device ML framework. Echodo does not talk to MLX directly — use an OpenAI-compatible HTTP server in front of MLX, same provider_type: openai_compatible as Ollama.

Server	Package	Default port	When to use
mlx_lm.server	pip install mlx-lm	8080	Solo dev, single implementer, simplest path
mlx-openai-server	cubist38/mlx-openai-server	8000	Parallel implementer fan-out — continuous batching handles concurrent requests better than the reference server
mlx-lm-server (community)	0xDaizz/mlx-lm-server	8000	Alternative if mlx-openai-server doesn't fit

Parallel implementers note: When the orchestrator dispatches multiple role-implementer jobs at once (disjoint briefs per agent-pipeline slice_dependencies), a single local inference process can become the bottleneck. Prefer mlx-openai-server (or mlx-lm-server with batching) over the minimal mlx_lm.server when running 2+ implementers against the same Mac. Ollama can also serve parallel requests but MLX-native models (mlx-community/* on Hugging Face) often perform better on Apple Silicon unified memory. You may run both Ollama (:11434) and an MLX server (:8000 / :8080) as separate registry aliases and route roles accordingly.

alias: qwen-coder-mlx
provider_type: openai_compatible
model_id: mlx-community/Qwen2.5-Coder-32B-Instruct-4bit
base_url: http://192.168.1.42:8000/v1   # mlx-openai-server; use :8080/v1 for mlx_lm.server
api_key_env_var: MLX_API_KEY            # placeholder; local servers ignore it
metadata: { tier: local, runtime: mlx }

Operator setup (mlx-openai-server example):

# Machine-wide install (all repos): ~/mlx-inference/mlx-serve.sh
# See ~/mlx-inference/README.md
~/mlx-inference/mlx-serve.sh

# Or manual:
pip install mlx-lm mlx-openai-server
mlx-openai-server launch \
  --model-path mlx-community/Qwen2.5-Coder-32B-Instruct-4bit \
  --model-type lm \
  --host 0.0.0.0 \
  --port 8000

Networking: Bind 0.0.0.0 when Echodo (Coolify) or apps/worker on another host must reach the Mac over LAN. Restrict firewall to homelab subnet. Do not expose raw local inference to the public internet.

v1 registry: No provider_type: mlx — HTTP gateway only. Native MLX SDK in apps/worker is out of scope.

alias: claude-planning
provider_type: anthropic
model_id: claude-sonnet-4-20250514
api_key_env_var: ANTHROPIC_API_KEY
metadata: { tier: premium }

NVIDIA NIM

alias: nim-llama-70b
provider_type: openai_compatible
model_id: meta/llama-3.1-70b-instruct
base_url: https://integrate.api.nvidia.com/v1
api_key_env_var: NVIDIA_API_KEY

Gemini

alias: gemini-flash
provider_type: google
model_id: gemini-2.0-flash
api_key_env_var: GOOGLE_GENERATIVE_AI_API_KEY

LiteLLM gateway (optional)

alias: litellm-gateway
provider_type: openai_compatible
model_id: qwen2.5-coder:32b  # or per-role model_id in route table
base_url: http://host:4000/v1
api_key_env_var: LITELLM_MASTER_KEY

Default role routing (seed template)

Shipped when a workspace enables pipeline integration:

role_id	Suggested alias	tier
role-conductor	claude-planning	premium
role-architect	claude-planning	premium
role-ia-architect	claude-planning	premium
role-ux-reviewer	qwen-coder or qwen-coder-mlx	fast/local
role-implementer	qwen-coder or qwen-coder-mlx	fast/local
role-reviewer	qwen-coder or qwen-coder-mlx	fast/local
role-design-system-auditor	qwen-coder or qwen-coder-mlx	fast/local
role-a11y-auditor	qwen-coder or qwen-coder-mlx	fast/local
role-doc-writer	qwen-coder or qwen-coder-mlx	fast/local
gate-copilot	claude-planning	premium
chat-default	workspace choice	—

Operators remap aliases in Settings; seed is not immutable.

Convoy / brief integration

Convoy frontmatter uses registry aliases, not Cursor model slugs:

model_policy:
  roles:
    role-conductor: claude-planning
    role-architect: claude-planning
    role-implementer: qwen-coder
  escalate_to: claude-planning

Brief frontmatter:

recommended_model: qwen-coder
model_tier: fast

Importer/parser for convoy files is out of scope for the first epic; orchestrator reads frontmatter when dispatching.

API surface (tRPC)

Router: ai.models (or pipeline.models)

Procedure	Purpose
listProviders	Paginated; workspace-scoped
createProvider / updateProvider / deleteProvider	Admin-only
testProvider	One-token completion; returns latency + error detail
listRoleRoutes	All routes for workspace
upsertRoleRoute	Set role → alias; supports custom roleId
deleteRoleRoute	Remove custom role
resolveForRole	Debug: given roleId + optional convoy/brief context

UI

• /{workspaceSlug}/settings/ai/providers — provider table + add/edit dialog + test button
• /{workspaceSlug}/settings/ai/role-routing — role → alias table + add custom role
• Link from convoy/gate screens: “Model for this step: qwen-coder (change)”

Logging contract

Extend agent_runs (and future convoy_events) with:

• model_alias varchar nullable
• provider_type varchar nullable
• model_id varchar nullable
• model_tier varchar nullable

complete_task MCP tool accepts optional modelAlias / modelTier for close-out logging.

Security

• Provider CRUD: workspace admin only (same bar as updateWorkflowPrompt).
• testProvider runs server-side; rate-limit per workspace.
• Never return resolved API keys to the browser — only alias and env var name.
• Worker/orchestrator processes must have env vars for aliases they use.

Relationship to other ADRs / plans

Artifact	Relationship
docs/model-routing-policy.md	Tier guidance; Cursor-era slugs migrate to aliases
plans/Plan-agent-coordination	claim_task / complete_task log model fields
plans/Plan-automation-foundation	Implements this ADR (model registry epic)
Future orchestrator epic	Calls resolveModel on every dispatch
Future worker epic	Consumes ResolvedModel only

Migration from Cursor slugs

Legacy convoys may reference claude-4.6-opus-high-thinking or composer-2.5-fast. Optional alias map in workspace metadata:

{
  "legacy_model_aliases": {
    "claude-4.6-opus-high-thinking": "claude-planning",
    "composer-2.5-fast": "qwen-coder"
  }
}

Consequences

Positive

• Single place to change Claude vs Qwen vs NIM without code deploys.
• Orchestrator and worker stay provider-agnostic.
• Extensible roles support pipeline evolution.

Negative

• Operators must configure providers before automation runs (mitigated: test button + clear errors).
• Two sources of truth for convoy overrides (markdown + DB) until convoy lifecycle MCP lands — resolution order is explicit.

Open questions

1. Repo-level echodo.models.yml — mirror workspace config in git for bootstrap? Deferred; DB is v1 source.
2. Per-user overrides — deferred; workspace-level only for v1.
3. Budget caps per alias — defer to LiteLLM or Phase 4 workers.

References

• Agent-pipeline: docs/model-routing-policy.md, docs/role-reference.md
• Agent-pipeline v0.4 plan: convoy lifecycle + Echodo integration
• Vercel AI SDK provider packages in packages/ai
• MLX: ml-explore/mlx, mlx-lm SERVER.md, mlx-openai-server
• Free-tier discovery: cheahjs/free-llm-api-resources, mnfst/awesome-free-llm-apis