feat(worker): OpenHands session bootstrap + 5 Agent Skills
Phase 1: Add 5 SKILL.md files to .agents/skills/ - backlog-sync: plans/ markdown ↔ DB contract - convoy-lifecycle: status machine, gates, MCP tools, checkpoints - model-routing: role→model via OmniRoute profiles + registry - code-review-two-axis: Standards vs Spec parallel sub-agent review - tdd-red-green: TDD loop with seams Phase 2: OmniRoute operational docs - docs/omniroute-role-mapping.md: profile→role matrix, convoy examples - docs/omniroute-setup.md: step-by-step operator guide Phase 3: Worker OpenHands integration - apps/worker/src/session/load-skills.ts: loads .agents/skills/ - apps/worker/src/session/openhands-session.ts: agent-server REST adapter - apps/worker/src/executor/index.ts: createOpenHandsExecutor() factory - apps/worker/src/env.ts: WORKER_SESSION_TYPE, AGENT_SERVER_URL, SESSION_API_KEY - apps/worker/src/handle-job.ts: routes to OpenHands when configured - apps/worker/src/session/minimal-session.ts: injects skills into system prompt - packages/shared/src/worker/index.ts: re-export WorkerExecutorOptions
This commit is contained in:
Randall Stillwell
parent
f6028615e8
commit
3fc7d01585
15 changed files with 3469 additions and 9 deletions
297
.agents/skills/backlog-sync/SKILL.md
Normal file
297
.agents/skills/backlog-sync/SKILL.md
Normal file
|
|
@ -0,0 +1,297 @@
|
|||
---
|
||||
name: backlog-sync
|
||||
description: Work with the plans/ markdown backlog - understand the strict layout (Plan/Epic/Task hierarchy), parse frontmatter, sync to markdown_backlog_items, and maintain the DB ↔ .md contract. Use when creating or modifying backlog items, understanding the import process, or working with the markdown-backlog package.
|
||||
---
|
||||
|
||||
# Backlog Sync
|
||||
|
||||
The `plans/` directory is the **source of truth** for product work. It's a strict markdown hierarchy parsed into the `markdown_backlog_items` database table. This skill teaches you the layout rules, frontmatter contract, and how the markdown ↔ DB sync works.
|
||||
|
||||
## When to Activate
|
||||
|
||||
Use this skill when:
|
||||
- Creating or editing files under `plans/`
|
||||
- Working with `markdown_backlog_items` or `cursor_sync_mappings` tables
|
||||
- Running or debugging the markdown import (`pnpm --filter @tasks/database watch:markdown-backlog`)
|
||||
- Understanding how `agent_prompt` inheritance works (plan → epic → task)
|
||||
- Troubleshooting backlog sync issues
|
||||
- Implementing features that read/write backlog data
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 1. Directory Structure is Strict
|
||||
|
||||
The parser depends on this **exact** layout. Do not improvise.
|
||||
|
||||
```
|
||||
plans/
|
||||
Plan-<slug>/
|
||||
Plan-<slug>.md ← Plan overview
|
||||
Epic-<slug>/
|
||||
Epic-<slug>.md ← Epic definition
|
||||
Task-<slug>.md ← Executable work item
|
||||
Task-<another>.md
|
||||
```
|
||||
|
||||
**Key constraints:**
|
||||
- Slugs are lowercase-hyphenated (`feature-x`, not `Feature_X` or `featureX`)
|
||||
- Slugs are stable — renaming breaks Cursor sync mappings
|
||||
- One task per file
|
||||
- File names must match the pattern: `Plan-<slug>.md`, `Epic-<slug>.md`, `Task-<slug>.md`
|
||||
|
||||
### 2. Frontmatter Contract
|
||||
|
||||
Every Plan/Epic/Task markdown file must start with YAML frontmatter. The parser extracts these fields:
|
||||
|
||||
**Required across all kinds:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
kind: plan | epic | task
|
||||
slug: matches-the-filename-slug
|
||||
title: Human-readable title
|
||||
status: draft | ready | in_progress | blocked | done | cancelled
|
||||
priority: P0 | P1 | P2 | P3
|
||||
tenant_id: UUID or "global"
|
||||
updated_at: "YYYY-MM-DD"
|
||||
---
|
||||
```
|
||||
|
||||
**Plan-specific additions:**
|
||||
|
||||
```yaml
|
||||
cursor_plan_id: null # populated by Cursor sync
|
||||
```
|
||||
|
||||
**Epic-specific additions:**
|
||||
|
||||
```yaml
|
||||
plan_slug: parent-plan-slug
|
||||
cursor_epic_id: null
|
||||
```
|
||||
|
||||
**Task-specific additions:**
|
||||
|
||||
```yaml
|
||||
plan_slug: parent-plan-slug
|
||||
epic_slug: parent-epic-slug
|
||||
owner: unassigned | github-username
|
||||
cursor_todo_id: null
|
||||
agent_prompt: |
|
||||
Optional multi-line prompt for agents working on this task.
|
||||
Inherits from epic → plan if not set here.
|
||||
```
|
||||
|
||||
**What the parser does:**
|
||||
- Extracts frontmatter via regex: `/^---\r?\n([\s\S]*?)\r?\n---\r?\n?/`
|
||||
- Parses YAML (tolerates syntax errors → empty object)
|
||||
- Infers `kind` from filename if frontmatter omits it (`Task-*.md` → `task`)
|
||||
- Infers `slug` from filename if frontmatter omits it (strips `Plan-`/`Epic-`/`Task-` prefix)
|
||||
- Infers `plan_slug` from path if frontmatter omits it (`plans/Plan-foo/...` → `foo`)
|
||||
- Infers `epic_slug` from path for tasks if frontmatter omits it (`Epic-bar/Task-x.md` → `bar`)
|
||||
- Computes `contentHash` (SHA-256 of full file) for idempotency
|
||||
- Stores `workflowPrompt` from `agent_prompt:` frontmatter field
|
||||
|
||||
### 3. Agent Prompt Inheritance
|
||||
|
||||
Tasks can define `agent_prompt:` in frontmatter. If missing, the system inherits:
|
||||
|
||||
```
|
||||
Task (no agent_prompt) → Epic (check agent_prompt) → Plan (check agent_prompt) → Default
|
||||
```
|
||||
|
||||
The `resolveWorkflowPrompt` function in `packages/database/src/markdown-backlog/resolve-prompt.ts` implements this cascade. When an agent claims a task via MCP `claim_task`, the resolved prompt is returned as context.
|
||||
|
||||
**Default prompt** (when nothing is set):
|
||||
|
||||
```
|
||||
You are an AI assistant helping implement a software task.
|
||||
Follow the acceptance criteria and constraints in the task description.
|
||||
Run pnpm lint && pnpm type-check after code changes.
|
||||
```
|
||||
|
||||
### 4. Database Sync Flow
|
||||
|
||||
**Import (Markdown → DB):**
|
||||
|
||||
1. Run `pnpm --filter @tasks/database watch:markdown-backlog`
|
||||
2. Scanner walks `plans/` for `*.md` files matching the pattern
|
||||
3. Parser extracts frontmatter + body + contentHash
|
||||
4. Syncer upserts rows to `markdown_backlog_items` by `(workspace_id, repo_path)` unique key
|
||||
5. Hierarchical `parent_id` links are computed (task → epic → plan)
|
||||
|
||||
**Key columns in `markdown_backlog_items`:**
|
||||
|
||||
| Column | Source |
|
||||
|--------|--------|
|
||||
| `workspace_id` | Env var `MARKDOWN_BACKLOG_WORKSPACE_ID` |
|
||||
| `repo_path` | Relative posix path: `plans/Plan-foo/Epic-bar/Task-baz.md` |
|
||||
| `kind` | Frontmatter `kind:` or inferred from filename |
|
||||
| `slug` | Frontmatter `slug:` or inferred from filename |
|
||||
| `title` | Frontmatter `title:` or first `# Heading` or filename |
|
||||
| `body` | Markdown body after frontmatter |
|
||||
| `content_hash` | SHA-256 of full file |
|
||||
| `parent_id` | Computed by matching `plan_slug`/`epic_slug` to parent rows |
|
||||
| `workflow_prompt` | Frontmatter `agent_prompt:` (null if not set) |
|
||||
|
||||
**Export (DB → Markdown):**
|
||||
|
||||
Not yet fully automated. The `exportBacklogItemToMarkdown` function can generate markdown from a DB row, but writes are manual or triggered via tRPC `backlog.createPlan/createEpic/createTask`.
|
||||
|
||||
### 5. Don't Break the Contract
|
||||
|
||||
**What NOT to do:**
|
||||
|
||||
- Don't nest tasks under other tasks (only Plan → Epic → Task)
|
||||
- Don't put secrets or API keys in frontmatter (files are committed)
|
||||
- Don't hand-edit `cursor_plan_id`, `cursor_epic_id`, `cursor_todo_id` (Cursor owns these)
|
||||
- Don't rename slugs without understanding the Cursor sync impact (breaks mappings)
|
||||
- Don't mix `snake_case` and `camelCase` in frontmatter keys (parser expects `plan_slug`, `epic_slug`, `agent_prompt`)
|
||||
- Don't create files outside the `plans/Plan-*/` structure (parser ignores them)
|
||||
|
||||
**What TO do:**
|
||||
|
||||
- Match existing style (see `docs/templates/plan-template.md`, `epic-template.md`, `task-template.md`)
|
||||
- Keep tasks **bead-scale** (one agent session can complete or materially advance)
|
||||
- Cross-link with relative paths: tasks link to epic via `./Epic-<slug>.md`, to plan via `../Plan-<slug>.md`
|
||||
- Use `tenant_id: global` for repo-wide work that's not tenant-specific
|
||||
- Run `pnpm --filter @tasks/database watch:markdown-backlog` after adding/editing files to sync
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Creating a New Task
|
||||
|
||||
1. Navigate to the Epic folder: `plans/Plan-<plan>/Epic-<epic>/`
|
||||
2. Create `Task-<slug>.md` (lowercase-hyphenated slug)
|
||||
3. Copy frontmatter template from `docs/templates/task-template.md`
|
||||
4. Fill required fields: `kind: task`, `slug`, `title`, `status: draft`, `priority`, `plan_slug`, `epic_slug`, `tenant_id`, `updated_at`
|
||||
5. Write task body (## Description, ## Acceptance criteria, etc.)
|
||||
6. Run `pnpm --filter @tasks/database watch:markdown-backlog` to import
|
||||
7. Verify row exists: `SELECT * FROM markdown_backlog_items WHERE slug = '<slug>'`
|
||||
|
||||
### Adding an Agent Prompt to a Task
|
||||
|
||||
If the task needs custom context for agents:
|
||||
|
||||
```yaml
|
||||
---
|
||||
kind: task
|
||||
slug: implement-feature-x
|
||||
# ... other required fields ...
|
||||
agent_prompt: |
|
||||
This task extends the existing authentication system.
|
||||
Key constraints:
|
||||
- Use NextAuth v5 (not v4)
|
||||
- Follow patterns in apps/web/lib/auth.ts
|
||||
- Add tests for new session logic
|
||||
---
|
||||
```
|
||||
|
||||
When an agent calls MCP `claim_task` with this task's `backlogItemId`, the prompt is returned and injected into the session context.
|
||||
|
||||
### Checking Sync Status
|
||||
|
||||
```bash
|
||||
# Watch mode (auto-imports on file change)
|
||||
pnpm --filter @tasks/database watch:markdown-backlog
|
||||
|
||||
# One-shot import
|
||||
MARKDOWN_BACKLOG_WORKSPACE_ID=<uuid> node packages/database/dist/markdown-backlog-cli.js
|
||||
```
|
||||
|
||||
**What it logs:**
|
||||
- Created: new rows inserted
|
||||
- Updated: contentHash changed
|
||||
- Skipped: contentHash matches (idempotent)
|
||||
- Errors: parse failures, missing parent refs
|
||||
|
||||
### Understanding the Hierarchy via SQL
|
||||
|
||||
```sql
|
||||
-- Get full tree for a plan
|
||||
WITH RECURSIVE tree AS (
|
||||
SELECT id, kind, slug, title, parent_id, 0 AS depth
|
||||
FROM markdown_backlog_items
|
||||
WHERE kind = 'plan' AND slug = 'automation-foundation'
|
||||
UNION ALL
|
||||
SELECT c.id, c.kind, c.slug, c.title, c.parent_id, p.depth + 1
|
||||
FROM markdown_backlog_items c
|
||||
JOIN tree p ON c.parent_id = p.id
|
||||
)
|
||||
SELECT kind, slug, title, depth FROM tree ORDER BY depth, slug;
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Valid Task Frontmatter
|
||||
|
||||
```yaml
|
||||
---
|
||||
kind: task
|
||||
slug: db-convoy-events-migration
|
||||
title: Drizzle convoy_events table and migration
|
||||
plan_slug: automation-foundation
|
||||
epic_slug: convoy-lifecycle-mcp
|
||||
status: done
|
||||
priority: P1
|
||||
tenant_id: global
|
||||
owner: unassigned
|
||||
cursor_todo_id: null
|
||||
updated_at: "2026-08-12"
|
||||
---
|
||||
|
||||
# Task summary
|
||||
|
||||
Add Drizzle schema + migration for convoy_events...
|
||||
```
|
||||
|
||||
### Example 2: Task with Agent Prompt
|
||||
|
||||
```yaml
|
||||
---
|
||||
kind: task
|
||||
slug: trpc-backlog-tree
|
||||
title: tRPC backlog.list / tree / get for markdown_backlog_items
|
||||
plan_slug: automation-foundation
|
||||
epic_slug: plans-roadmap-ui
|
||||
status: ready
|
||||
priority: P1
|
||||
tenant_id: global
|
||||
owner: unassigned
|
||||
cursor_todo_id: null
|
||||
updated_at: "2026-08-12"
|
||||
agent_prompt: |
|
||||
Extend apps/web backlog router: list/tree/get for markdown_backlog_items.
|
||||
Always filter by workspace_id. Zod inputs. No Plans UI yet — API only.
|
||||
Reuse existing schema. claim_task / complete_task; lint + type-check + test.
|
||||
---
|
||||
```
|
||||
|
||||
### Example 3: Debugging Import Issues
|
||||
|
||||
**Problem:** Task not appearing in DB after markdown edit.
|
||||
|
||||
**Diagnosis:**
|
||||
1. Check file structure: `ls plans/Plan-<plan>/Epic-<epic>/Task-<slug>.md`
|
||||
2. Validate frontmatter YAML syntax (common issue: unbalanced quotes, missing colons)
|
||||
3. Check watcher logs: `pnpm --filter @tasks/database watch:markdown-backlog` (look for parse errors)
|
||||
4. Query DB: `SELECT * FROM markdown_backlog_items WHERE slug = '<slug>' AND workspace_id = '<uuid>'`
|
||||
5. Check `content_hash`: if it matches, sync is idempotent (no change detected)
|
||||
|
||||
**Fix:** Correct frontmatter YAML, re-run import, verify row updated.
|
||||
|
||||
## References
|
||||
|
||||
- **Parser source:** `packages/database/src/markdown-backlog/parse.ts`
|
||||
- **Sync logic:** `packages/database/src/markdown-backlog/sync.ts`
|
||||
- **Prompt resolution:** `packages/database/src/markdown-backlog/resolve-prompt.ts`
|
||||
- **Templates:** `docs/templates/plan-template.md`, `epic-template.md`, `task-template.md`
|
||||
- **Glossary:** `docs/Glossary.md` (backlog hierarchy, sync vocabulary)
|
||||
- **Cursor sync notes:** `config/CursorSync.md` (push/pull behavior, mappings)
|
||||
- **Schema:** `packages/database/src/schema/markdown_backlog_items.ts`
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `convoy-lifecycle` — when backlog tasks map to convoy briefs
|
||||
- `model-routing` — how agent_prompt + model_policy interact
|
||||
- `code-review-two-axis` — reviewing changes to backlog sync code
|
||||
243
.agents/skills/code-review-two-axis/SKILL.md
Normal file
243
.agents/skills/code-review-two-axis/SKILL.md
Normal file
|
|
@ -0,0 +1,243 @@
|
|||
---
|
||||
name: code-review-two-axis
|
||||
description: Review code changes along two axes: Standards (does code follow repo conventions?) and Spec (does it match the originating issue/requirement?). Uses parallel sub-agents to keep concerns separate, then aggregates findings. Use when role-reviewer runs, a human requests PR review, or you're asked to review "changes since X".
|
||||
---
|
||||
|
||||
# Code Review: Two-Axis Method
|
||||
|
||||
A rigorous but practical code review method that separates **Standards** from **Spec**, runs them as parallel sub-agents, and aggregates findings. Adapted from the approach in [mattpocock/skills](https://github.com/mattpocock/skills/blob/main/skills/engineering/code-review/SKILL.md) for Echodo's convoy pipeline.
|
||||
|
||||
## When to Activate
|
||||
|
||||
Use this skill when:
|
||||
- **Role-reviewer** is triggered during convoy `pr_review` gate
|
||||
- A human requests a PR review in Mission Control
|
||||
- Someone asks "review the changes since commit X" or "review branch Y"
|
||||
- You're asked to evaluate if code meets the acceptance criteria from the originating brief
|
||||
|
||||
## Core Method
|
||||
|
||||
### Two Axes
|
||||
|
||||
**1. Standards (Does the code follow repo conventions?)**
|
||||
- Follows `AGENTS.md` and codebase style
|
||||
- Uses existing patterns (file locations, naming, imports)
|
||||
- Passes linter/formatter checks
|
||||
- No obvious code smells (from Fowler's refactoring book)
|
||||
|
||||
**2. Spec (Does the code match what was asked for?)**
|
||||
- Implements acceptance criteria from the originating brief
|
||||
- Matches the description in `.convoys/<slug>/brief-*.md`
|
||||
- Addresses the issue/story/task it claims to solve
|
||||
- No missing functionality or edge cases
|
||||
|
||||
### Why Two Separate?
|
||||
|
||||
- **Separation of concerns:** A change might be well-written (Standards) but not solve the problem (Spec), or vice versa
|
||||
- **Clearer feedback:** "This is good code but doesn't match the spec" vs "This matches the spec but violates our style"
|
||||
- **Parallel sub-agents:** Run both in parallel to avoid context pollution between concerns
|
||||
|
||||
## The Process
|
||||
|
||||
### Step 1: Pin the Fixed Point
|
||||
|
||||
Whatever the user specifies as the comparison base:
|
||||
- Commit SHA: `abc1234`
|
||||
- Branch: `feature-x`
|
||||
- Tag: `v1.2.0`
|
||||
- Relative: `HEAD~5`, `main`
|
||||
|
||||
**Capture:**
|
||||
- Diff command: `git diff <fixed-point>...HEAD` (three-dot for merge-base comparison)
|
||||
- Commit log: `git log <fixed-point>..HEAD --oneline`
|
||||
|
||||
**Validate:**
|
||||
- `git rev-parse <fixed-point>` must resolve
|
||||
- Diff must be non-empty
|
||||
|
||||
### Step 2: Identify the Spec Source
|
||||
|
||||
Find what the change is supposed to implement, in this order:
|
||||
|
||||
1. **Issue references in commits:** `#123`, `Closes #45`, etc.
|
||||
2. **Brief path:** User passed `.convoys/<slug>/brief-N-*.md`
|
||||
3. **Matching spec file:** Look in `docs/`, `specs/`, `.scratch/` for files matching branch name
|
||||
4. **Ask the user:** If none found, ask where the spec is
|
||||
|
||||
### Step 3: Identify the Standards Sources
|
||||
|
||||
- `AGENTS.md` at repo root
|
||||
- `.cursor/rules/*.mdc` files (if not migrated to `.agents/skills/`)
|
||||
- `docs/` for architecture decisions
|
||||
- Existing code patterns in the same module
|
||||
- **Fowler code smells baseline** (below)
|
||||
|
||||
### Step 4: Run Both Sub-Agents in Parallel
|
||||
|
||||
Run **Standards agent** and **Spec agent** as separate conversations with their own context. Each receives:
|
||||
- The diff
|
||||
- Its specific criteria
|
||||
- Output format instructions
|
||||
|
||||
**Standards sub-agent receives:**
|
||||
- Diff
|
||||
- Standards sources (AGENTS.md, rules, patterns)
|
||||
- Code smell baseline
|
||||
- Instruction: "Review ONLY for standards violations"
|
||||
|
||||
**Spec sub-agent receives:**
|
||||
- Diff
|
||||
- Brief/spec source
|
||||
- Acceptance criteria from originating task
|
||||
- Instruction: "Review ONLY for spec conformance"
|
||||
|
||||
### Step 5: Aggregate Findings
|
||||
|
||||
After both sub-agents complete, merge their reports:
|
||||
|
||||
```markdown
|
||||
# Code Review: <convoy/branch>
|
||||
|
||||
## Summary
|
||||
<Combined 2-3 sentence summary>
|
||||
|
||||
## Standards Findings
|
||||
<Issues from Standards axis, grouped by severity>
|
||||
|
||||
## Spec Findings
|
||||
<Issues from Spec axis, grouped by severity>
|
||||
|
||||
## Recommendations
|
||||
<Combined recommendations, prioritized>
|
||||
|
||||
## Verdict
|
||||
[ ] Approve - meets standards and spec
|
||||
[ ] Approve with comments - minor issues, not blocking
|
||||
[ ] Request changes - significant issues must be addressed
|
||||
[ ] Block - blocker-level issues (security, correctness)
|
||||
```
|
||||
|
||||
## Code Smell Baseline (Standards)
|
||||
|
||||
Even if repo docs say nothing, apply this baseline. **Repo rules always override.**
|
||||
|
||||
### The 11 Smells
|
||||
|
||||
| Smell | What It Is | How to Fix |
|
||||
|-------|------------|------------|
|
||||
| **Mysterious Name** | Function/variable/type name doesn't reveal purpose | Rename to something descriptive |
|
||||
| **Duplicated Code** | Same logic in multiple files/hunks | Extract to shared function |
|
||||
| **Feature Envy** | Method reaches into another object's data more than its own | Move method to the data it envys |
|
||||
| **Data Clumps** | Same fields/params travel together | Bundle into a type |
|
||||
| **Primitive Obsession** | Primitives standing in for domain concepts | Create small types |
|
||||
| **Repeated Switches** | Same switch/if-cascade repeats | Use polymorphism or map |
|
||||
| **Shotgun Surgery** | One logical change forces edits in many files | Gather related changes together |
|
||||
| **Divergent Change** | One file edited for unrelated reasons | Split by concern |
|
||||
| **Speculative Generality** | Abstraction/params added for imagined future needs | Delete until real need appears |
|
||||
| **Message Chains** | Long `a.b().c().d()` navigation | Hide behind method on first object |
|
||||
| **Middle Man** | Class/function that mostly just delegates | Call target directly |
|
||||
| **Refused Bequest** | Subclass ignores most of what it inherits | Use composition, not inheritance |
|
||||
|
||||
## Severity Levels
|
||||
|
||||
| Level | Meaning | Action |
|
||||
|-------|---------|--------|
|
||||
| **Blocker** | Security vulnerability, correctness bug, data loss risk | Must fix before merge |
|
||||
| **Major** | Significant issue affecting functionality or maintainability | Should fix before merge |
|
||||
| **Minor** | Nits, style issues, minor improvements | Fix before merge or follow-up |
|
||||
| **Nit** | Style preference, whitespace, naming convention | Optional, can defer |
|
||||
|
||||
## Categories
|
||||
|
||||
| Category | What It Covers |
|
||||
|----------|----------------|
|
||||
| **Correctness** | Logic errors, bugs, edge cases not handled |
|
||||
| **Security** | Vulnerabilities, input validation, auth issues |
|
||||
| **Performance** | Inefficiency, unnecessary work, memory issues |
|
||||
| **Readability** | Clarity, naming, documentation |
|
||||
| **Style** | Formatting, lint violations (let linter handle these) |
|
||||
| **Test-Coverage** | Missing tests for new functionality |
|
||||
|
||||
## Example Review Output
|
||||
|
||||
### Input
|
||||
|
||||
- **Convoy:** `feature-add-auth-endpoint`
|
||||
- **Fixed point:** `main`
|
||||
- **Brief:** `.convoys/feature-add-auth-endpoint/brief-1-implement.md`
|
||||
- **Acceptance criteria from brief:**
|
||||
- [ ] Route at `apps/web/app/api/auth/register/route.ts`
|
||||
- [ ] zod schema validates email/password
|
||||
- [ ] Database insert creates user row
|
||||
- [ ] Returns 201 on success, 400 on validation error
|
||||
|
||||
### Output
|
||||
|
||||
```markdown
|
||||
# Code Review: feature-add-auth-endpoint
|
||||
|
||||
## Summary
|
||||
Implements POST /api/auth/register endpoint with zod validation and database insert. Route file correctly placed. Meets most acceptance criteria. One validation edge case and missing test coverage identified.
|
||||
|
||||
## Standards Findings
|
||||
|
||||
### Minor
|
||||
- **[style]** `apps/web/app/api/auth/register/route.ts:15` — Consider extracting validation schema to `lib/validations/auth.ts` for reusability
|
||||
- **[readability]** `apps/web/app/api/auth/register/route.ts:8` — `createUser` function name could be more descriptive: `createUserFromCredentials`
|
||||
|
||||
### Nit
|
||||
- **[style]** Missing blank line between imports
|
||||
|
||||
## Spec Findings
|
||||
|
||||
### Major
|
||||
- **[correctness]** `apps/web/app/api/auth/register/route.ts:28` — Returns 500 on database error instead of appropriate 4xx. Should return 409 if email already exists.
|
||||
|
||||
### Minor
|
||||
- **[test-coverage]** No test file created for this route. Brief AC didn't explicitly require tests but Echodo standard is to add tests.
|
||||
|
||||
## Recommendations
|
||||
|
||||
1. **High priority:** Handle duplicate email (409) and database errors (500 → 500, but logged)
|
||||
2. **Medium:** Consider extracting validation schema
|
||||
3. **Low:** Add minimal test coverage
|
||||
|
||||
## Verdict
|
||||
|
||||
[ ] Approve
|
||||
[X] Approve with comments - one major issue should be addressed
|
||||
[ ] Request changes
|
||||
[ ] Block
|
||||
|
||||
## Metadata
|
||||
- **Reviewed by:** role-reviewer (qwen-coder-mlx)
|
||||
- **Model:** qwen-coder-mlx
|
||||
- **Files reviewed:** 1
|
||||
- **Lines changed:** ~45
|
||||
```
|
||||
|
||||
## Running as Role-Reviewer
|
||||
|
||||
When triggered as `role-reviewer` in a convoy:
|
||||
|
||||
1. **Find the brief:** Look at convoy's current brief in `.convoys/<slug>/brief-*.md`
|
||||
2. **Extract AC:** Get acceptance criteria from brief body
|
||||
3. **Pin base:** Use `main` as default fixed point, or ask if different
|
||||
4. **Run parallel agents:** Standards + Spec in parallel
|
||||
5. **Aggregate:** Merge into final review per format above
|
||||
6. **Log:** Call `log_convoy_event` with event type `role_complete`, actor `role-reviewer`
|
||||
7. **Transition:** If blocker/major issues, transition convoy back to `in_progress`. If minor/nits, transition to `pr_review`.
|
||||
|
||||
## Interaction with Other Skills
|
||||
|
||||
- **backlog-sync:** When review involves changes to markdown import logic
|
||||
- **convoy-lifecycle:** When logging role_complete and transitioning status
|
||||
- **model-routing:** When determining which model runs the review
|
||||
- **tdd-red-green:** When suggesting tests that should exist
|
||||
|
||||
## References
|
||||
|
||||
- **Original skill:** https://github.com/mattpocock/skills/blob/main/skills/engineering/code-review/SKILL.md
|
||||
- **Fowler Refactoring:** Chapter 3 - "Bad Smells in Code"
|
||||
- **AGENTS.md:** Root of repo for Echodo-specific standards
|
||||
- **Echodo patterns:** Use existing code in `apps/web/`, `packages/` as standards reference
|
||||
440
.agents/skills/convoy-lifecycle/SKILL.md
Normal file
440
.agents/skills/convoy-lifecycle/SKILL.md
Normal file
|
|
@ -0,0 +1,440 @@
|
|||
---
|
||||
name: convoy-lifecycle
|
||||
description: Work with the convoy system - understand status transitions, gate actors, MCP tools, and checkpoint/resume. Use when creating/managing convoys, transitioning status, logging events, or implementing the worker harness.
|
||||
---
|
||||
|
||||
# Convoy Lifecycle
|
||||
|
||||
Convoys are the pipeline execution model for Echodo. A convoy groups related briefs into a cohesive unit with a success metric, model policy, and human gates. This skill teaches you the status machine, actor rules, MCP tools, and how the orchestrator drives the flow.
|
||||
|
||||
## When to Activate
|
||||
|
||||
Use this skill when:
|
||||
- Creating or managing `.convoys/*.md` files
|
||||
- Transitioning convoy status (draft → ready → merged, etc.)
|
||||
- Understanding gate actors and what they can/cannot do
|
||||
- Implementing the worker harness (Epic 7)
|
||||
- Logging events to `convoy_events` or `agent_runs`
|
||||
- Checking checkpoint/resume behavior
|
||||
- Debugging convoy state issues
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 1. Status Machine (The Contract)
|
||||
|
||||
Convoys have a strict status lifecycle. Transitions are enforced by the server.
|
||||
|
||||
**Code flow (from → to):**
|
||||
|
||||
```
|
||||
draft ──→ convoy_review ──→ ready ──→ in_progress ──→ pr_review ──→ merged ──→ released
|
||||
↑ │ ↑
|
||||
│ │ │
|
||||
│ └──→ blocked └──→ blocked
|
||||
│ ↑
|
||||
└──→ cancelled └──→ cancelled
|
||||
```
|
||||
|
||||
**Non-code variant (docs/SOPs):**
|
||||
|
||||
```
|
||||
draft ──→ convoy_review ──→ ready ──→ in_progress ──→ doc_review ──→ published ──→ released
|
||||
↑ │ ↑
|
||||
│ └──→ blocked └──→ blocked
|
||||
│
|
||||
└──→ cancelled
|
||||
```
|
||||
|
||||
**Terminal states:** `released`, `cancelled`
|
||||
|
||||
**Exceptional state:** `blocked` (can resume from most states)
|
||||
|
||||
### 2. Actor Rules (Who Can Do What)
|
||||
|
||||
The `ALLOWED_TRANSITIONS` table in `packages/shared/src/convoy-lifecycle/state-machine.ts` defines explicit actor permissions. **Pipeline roles cannot perform gate transitions.**
|
||||
|
||||
| Actor | Can Do |
|
||||
|-------|--------|
|
||||
| `role-architect` | Submit draft to convoy_review |
|
||||
| `role-conductor` | Cancel convoy, unblock from blocked |
|
||||
| `role-implementer` | Submit ready to in_progress, move pr_review back to in_progress, block/unblock |
|
||||
| `human-gate-1` | Approve/reject convoy_review → ready, block/unblock, cancel |
|
||||
| `human-gate-2` | Approve pr_review → merged, doc_review → published, block/unblock, cancel |
|
||||
| `human-gate-3` | Approve merged → released |
|
||||
| `doc-approver` | Alias for human-gate-2 on doc flows |
|
||||
| `automation` | Orchestrator-driven transitions (checkpoint resume, stall recovery) |
|
||||
|
||||
**Key principle:** `role-architect` ≠ human-gate-1. The conductor/architect roles plan and kick off; humans approve and release.
|
||||
|
||||
### 3. Gate Actors are Not Pipeline Roles
|
||||
|
||||
This is a common mistake. Do NOT let pipeline roles perform gates:
|
||||
|
||||
- ❌ `role-implementer` cannot approve PR merge (must be `human-gate-2`)
|
||||
- ❌ `role-architect` cannot release to production (must be `human-gate-3`)
|
||||
- ❌ `role-conductor` cannot approve convoy_review (must be `human-gate-1`)
|
||||
|
||||
**Exception:** The orchestrator runs as `automation` actor for internal transitions (checkpoint resume, stall recovery).
|
||||
|
||||
### 4. Reason Requirement
|
||||
|
||||
Some transitions require a `reason` string (e.g., blocking, cancelling, unblocking):
|
||||
|
||||
```typescript
|
||||
// From state-machine.ts
|
||||
{ from: "blocked", to: "ready", actors: ["human-gate-1", "role-conductor"], requiresReason: true }
|
||||
```
|
||||
|
||||
When calling MCP `transition_convoy_status` for these, pass `reason: "blocking for redesign"`.
|
||||
|
||||
### 5. File-Based Convoys (`.convoys/*.md`)
|
||||
|
||||
Convoys are defined in markdown with YAML frontmatter. Location: `.convoys/<slug>.md`.
|
||||
|
||||
**Frontmatter structure:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: <slug>
|
||||
classification: server-only | web | full-stack
|
||||
success_metric: One sentence explaining what "done" looks like
|
||||
skip:
|
||||
- ia # skip IA review
|
||||
- ux # skip UX review
|
||||
- design
|
||||
- a11y
|
||||
- visual
|
||||
- smoke
|
||||
status: draft | in_progress | merged | shipped | abandoned
|
||||
created: YYYY-MM-DD
|
||||
model_policy:
|
||||
roles:
|
||||
role-conductor: claude-planning
|
||||
role-architect: claude-planning
|
||||
role-implementer: qwen-coder
|
||||
role-reviewer: qwen-coder
|
||||
escalate_to: claude-planning
|
||||
---
|
||||
```
|
||||
|
||||
**Body sections:**
|
||||
- **Why:** Problem statement and motivation
|
||||
- **Scope:** What's in/out
|
||||
- **Roles invoked:** Who runs which phases
|
||||
- **Todos:** Brief mapping
|
||||
- **Architecture:** Technical notes
|
||||
|
||||
**Briefs live in:** `.convoys/<slug>/brief-<N>-<title>.md`
|
||||
|
||||
### 6. Checkpoint/Resume (Worker Feature)
|
||||
|
||||
When a worker job runs, it can checkpoint state to `worker_checkpoints`:
|
||||
|
||||
```typescript
|
||||
// Checkpoint payload (example from docs/worker.md)
|
||||
{
|
||||
toolHistory: [{ tool: "read_file", input: { path: "..." }, output: "..." }],
|
||||
status: "in_progress",
|
||||
lastCheckpointAt: "2026-08-12T10:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**Resume behavior:**
|
||||
- Worker kills → orchestrator detects stall (`ORCHESTRATOR_STALL_MINUTES` default 120)
|
||||
- Next dequeue reads checkpoint for same job → resumes from where it left off
|
||||
- `toolHistory` enables context replay without re-running everything
|
||||
|
||||
### 7. Events Table (`convoy_events`)
|
||||
|
||||
Every transition and role completion is logged to `convoy_events`:
|
||||
|
||||
```sql
|
||||
-- Example log entries
|
||||
| id | convoy_slug | event_type | actor | metadata |
|
||||
|----|-------------|-----------------|---------------|-------------------------------|
|
||||
| 1 | feature-x | status_change | role-architect| { from: "draft", to: "ready" }|
|
||||
| 2 | feature-x | role_complete | role-architect| { brief: "brief-1", ... } |
|
||||
| 3 | feature-x | role_complete | role-implementer| { brief: "brief-2", model: ...}|
|
||||
| 4 | feature-x | status_change | human-gate-2 | { from: "pr_review", to: "merged" } |
|
||||
```
|
||||
|
||||
**Event types:**
|
||||
- `status_change` — transition from one convoy status to another
|
||||
- `role_complete` — a brief finished with metrics (model, tokens, duration)
|
||||
- `stall_recovered` — orchestrator resumed from checkpoint
|
||||
- `gate_action` — human approved/requested changes
|
||||
|
||||
## MCP Tools
|
||||
|
||||
### create_convoy
|
||||
|
||||
**Purpose:** Create a new convoy at `draft` status with metadata.
|
||||
|
||||
**Signature:**
|
||||
```typescript
|
||||
create_convoy({
|
||||
workspace: string, // Workspace UUID or slug
|
||||
slug: string, // Convoy slug (kebab-case)
|
||||
title: string,
|
||||
successMetric: string, // Frontmatter success_metric
|
||||
classification: string, // server-only | web | full-stack
|
||||
skip?: string[], // Optional: ["ia", "ux", "design"]
|
||||
modelPolicy?: { // Optional: set model routes
|
||||
roles: Record<string, string>, // role -> alias
|
||||
escalateTo?: string,
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Returns:** Convoy object with ID, slug, current status (`draft`), created_at.
|
||||
|
||||
**Example usage:**
|
||||
```
|
||||
"Create a convoy called 'test-omniroute-integration' with success metric 'OmniRoute routes all 5 roles successfully'. Classification: server-only. Skip IA, UX, design."
|
||||
```
|
||||
|
||||
### transition_convoy_status
|
||||
|
||||
**Purpose:** Move convoy between statuses. Enforces actor rules + reason requirements.
|
||||
|
||||
**Signature:**
|
||||
```typescript
|
||||
transition_convoy_status({
|
||||
workspace: string,
|
||||
convoySlug: string,
|
||||
fromStatus: string,
|
||||
toStatus: string,
|
||||
actor: string, // role-architect, human-gate-1, etc.
|
||||
reason?: string, // Required for block/unblock/cancel
|
||||
})
|
||||
```
|
||||
|
||||
**Returns:** Updated convoy status, event log entry.
|
||||
|
||||
**Example usage:**
|
||||
```
|
||||
"Transition convoy 'test-e2e-autonomous' from 'ready' to 'in_progress'. Actor: role-implementer."
|
||||
```
|
||||
|
||||
### log_convoy_event
|
||||
|
||||
**Purpose:** Append a role completion or custom event to `convoy_events`.
|
||||
|
||||
**Signature:**
|
||||
```typescript
|
||||
log_convoy_event({
|
||||
workspace: string,
|
||||
convoySlug: string,
|
||||
eventType: string, // role_complete, gate_action, etc.
|
||||
actor: string, // Who performed the event
|
||||
metadata: Record<string, unknown>, // Additional context
|
||||
})
|
||||
```
|
||||
|
||||
**Example usage:**
|
||||
```
|
||||
"Log role_complete for convoy 'test-e2e-autonomous', actor: role-implementer, brief: brief-2-implement-feature, model: qwen-coder, tokens: 12345"
|
||||
```
|
||||
|
||||
### query_manifest_status
|
||||
|
||||
**Purpose:** Get convoy status, briefs, and recent events for Mission Control UI.
|
||||
|
||||
**Signature:**
|
||||
```typescript
|
||||
query_manifest_status({
|
||||
workspace: string,
|
||||
convoySlug?: string, // Optional: filter to one convoy
|
||||
})
|
||||
```
|
||||
|
||||
**Returns:** Convoy object with status, briefs (with status), recent events.
|
||||
|
||||
### reconcile_from_files
|
||||
|
||||
**Purpose:** Import `.convoys/*.md` into the DB and report pending sync outbox.
|
||||
|
||||
**Signature:**
|
||||
```typescript
|
||||
reconcile_from_files({
|
||||
repoPath?: string, // Optional: scan specific path
|
||||
})
|
||||
```
|
||||
|
||||
**Returns:** Summary of created/updated convoys, any sync errors.
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Creating a Convoy
|
||||
|
||||
1. **Define the convoy file:** `.convoys/feature-x.md`
|
||||
2. **Write frontmatter:** name, classification, success_metric, model_policy
|
||||
3. **Write body:** Why, Scope, Roles, Todos
|
||||
4. **Create briefs:** `.convoys/feature-x/brief-1-*.md` for each task
|
||||
5. **Import to DB:** `reconcile_from_files` or `create_convoy` MCP
|
||||
6. **Start work:** `transition_convoy_status` to `convoy_review`
|
||||
|
||||
### Running Through Gates
|
||||
|
||||
1. **Draft → Convoy Review:** `role-architect` submits (or `create_convoy` with status)
|
||||
2. **Convoy Review:** `human-gate-1` approves (→ ready) or rejects (→ draft)
|
||||
3. **Ready → In Progress:** `role-implementer` starts work
|
||||
4. **In Progress → PR Review:** `role-implementer` submits code
|
||||
5. **PR Review:** `human-gate-2` approves (→ merged) or requests changes (→ in_progress)
|
||||
6. **Merged → Released:** `human-gate-3` promotes to production
|
||||
|
||||
### Logging Role Completion
|
||||
|
||||
When an agent finishes a brief:
|
||||
|
||||
```typescript
|
||||
// In worker, after complete_task MCP
|
||||
await log_convoy_event({
|
||||
workspace: workspaceId,
|
||||
convoySlug: convoy.slug,
|
||||
eventType: "role_complete",
|
||||
actor: roleId, // e.g., "role-implementer"
|
||||
metadata: {
|
||||
briefSlug: brief.slug,
|
||||
model: modelSpec.alias, // e.g., "qwen-coder"
|
||||
modelTier: tier, // "coding" or "coding-pro"
|
||||
tokensIn: tokens.input,
|
||||
tokensOut: tokens.output,
|
||||
durationMs: duration,
|
||||
outcome: "success" | "failure",
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### Handling Blocks
|
||||
|
||||
To block a convoy (any gate actor or role-implementer):
|
||||
|
||||
```typescript
|
||||
await transition_convoy_status({
|
||||
workspace: workspaceId,
|
||||
convoySlug: "feature-x",
|
||||
fromStatus: "in_progress",
|
||||
toStatus: "blocked",
|
||||
actor: "human-gate-2", // or role-implementer
|
||||
reason: "Design changed - need API redesign before continuing"
|
||||
});
|
||||
```
|
||||
|
||||
To unblock:
|
||||
|
||||
```typescript
|
||||
await transition_convoy_status({
|
||||
workspace: workspaceId,
|
||||
convoySlug: "feature-x",
|
||||
fromStatus: "blocked",
|
||||
toStatus: "in_progress",
|
||||
actor: "human-gate-2", // or role-conductor
|
||||
reason: "API redesign complete"
|
||||
});
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Simple Code Convoy
|
||||
|
||||
**`.convoys/add-auth-endpoint.md`:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: add-auth-endpoint
|
||||
classification: server-only
|
||||
success_metric: POST /api/auth/register returns 201 and creates user row
|
||||
skip:
|
||||
- ia
|
||||
- ux
|
||||
- design
|
||||
status: draft
|
||||
created: 2026-08-12
|
||||
model_policy:
|
||||
roles:
|
||||
role-architect: claude-planning
|
||||
role-implementer: qwen-coder
|
||||
role-reviewer: qwen-coder
|
||||
---
|
||||
```
|
||||
|
||||
**Brief:** `.convoys/add-auth-endpoint/brief-1-implement.md`
|
||||
|
||||
```markdown
|
||||
## Description
|
||||
|
||||
Implement the POST /api/auth/register endpoint.
|
||||
|
||||
## Scope
|
||||
|
||||
- Create route handler at apps/web/app/api/auth/register/route.ts
|
||||
- Add validation with zod
|
||||
- Call database insert
|
||||
- Return 201 on success, 400 on validation error
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Route file exists at correct path
|
||||
- [ ] zod schema validates email/password
|
||||
- [ ] Database insert creates user row
|
||||
- [ ] pnpm lint && pnpm type-check pass
|
||||
- [ ] Brief is done
|
||||
```
|
||||
|
||||
### Example 2: Doc-Only Convoy
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: write-sops
|
||||
classification: server-only
|
||||
success_metric: 5 SOP documents are published under docs/sops/
|
||||
skip:
|
||||
- ia
|
||||
- ux
|
||||
- design
|
||||
- visual
|
||||
- smoke
|
||||
status: draft
|
||||
created: 2026-08-12
|
||||
model_policy:
|
||||
roles:
|
||||
role-doc-writer: auto/coding
|
||||
---
|
||||
|
||||
Use the template at docs/templates/doc-sop.md to write SOPs for:
|
||||
- How to run a conveyor
|
||||
- How to add a new model provider
|
||||
- How to configure OmniRoute
|
||||
- How to add a new harness
|
||||
```
|
||||
|
||||
### Example 3: Status Transition Flow
|
||||
|
||||
```
|
||||
1. Architect submits: transition(ready → convoy_review) by role-architect
|
||||
2. Gate approves: transition(convoy_review → ready) by human-gate-1
|
||||
3. Implementer starts: transition(ready → in_progress) by role-implementer
|
||||
4. Implementer submits: transition(in_progress → pr_review) by role-implementer
|
||||
5. Gate merges: transition(pr_review → merged) by human-gate-2
|
||||
6. Gate releases: transition(merged → released) by human-gate-3
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- **State machine:** `packages/shared/src/convoy-lifecycle/state-machine.ts`
|
||||
- **Schemas:** `packages/shared/src/convoy-lifecycle/schemas.ts`
|
||||
- **Kanban view:** `packages/shared/src/convoy-lifecycle/kanban.ts`
|
||||
- **Tests:** `packages/shared/src/convoy-lifecycle/state-machine.test.ts`
|
||||
- **MCP tools:** `apps/mcp-server/src/tools/create-convoy.ts`, `transition_convoy_status.ts`, `log_convoy_event.ts`
|
||||
- **Worker docs:** `docs/worker.md`
|
||||
- **Orchestrator docs:** `docs/orchestrator.md`
|
||||
- **Convoy lifecycle doc:** `docs/convoy-lifecycle.md`
|
||||
- **Example convoys:** `.convoys/second-brain-os-v01.md`, `.convoys/polish-dag-tiers-cost.md`
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `backlog-sync` — when briefs map to markdown_backlog_items
|
||||
- `model-routing` — how model_policy aliases resolve via omniroute
|
||||
- `openhands-worker` — worker implementation that uses this lifecycle
|
||||
- `code-review-two-axis` — what role-reviewer does during pr_review
|
||||
463
.agents/skills/model-routing/SKILL.md
Normal file
463
.agents/skills/model-routing/SKILL.md
Normal file
|
|
@ -0,0 +1,463 @@
|
|||
---
|
||||
name: model-routing
|
||||
description: Route pipeline roles to AI models via workspace registry and OmniRoute profiles. Understand resolution order, convoy overrides, tier mapping, and OmniRoute integration. Use when configuring models, debugging route failures, or working with model_policy.
|
||||
---
|
||||
|
||||
# Model Routing
|
||||
|
||||
Echodo's model routing system connects **pipeline roles** (conductor, architect, implementer, reviewer) to **AI models** via a workspace-scoped registry with layered overrides. The system integrates with OmniRoute for dynamic model selection across 5 profiles (`auto`, `auto/coding`, `auto/coding-pro`, `auto/reasoning`, `auto/reasoning-pro`). This skill teaches you how routes resolve, where to configure them, and how the orchestrator dispatches work.
|
||||
|
||||
## When to Activate
|
||||
|
||||
Use this skill when:
|
||||
- Configuring model providers or role routes in Settings → AI
|
||||
- Understanding `model_policy` in `.convoys/*.md`
|
||||
- Debugging model resolution failures
|
||||
- Integrating OmniRoute (cloud or local)
|
||||
- Logging model metadata in `agent_runs` or `convoy_events`
|
||||
- Adding new pipeline roles that need model routing
|
||||
- Testing provider connectivity (`testProvider`)
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### 1. Four-Layer Resolution Order (First Match Wins)
|
||||
|
||||
**Priority from highest to lowest:**
|
||||
|
||||
```
|
||||
1. Brief override ──→ recommended_model in .convoys/<slug>/brief-<N>-*.md
|
||||
(Direct per-task model selection)
|
||||
|
||||
2. Convoy override ──→ model_policy.roles in .convoys/<slug>.md
|
||||
(Per-convoy role→alias mapping)
|
||||
|
||||
3. Workspace route ──→ role_model_routes row in DB
|
||||
(Settings → AI → Role Routing table)
|
||||
|
||||
4. Pipeline template ──→ default role mapping
|
||||
(Shipped seed data for new workspaces)
|
||||
```
|
||||
|
||||
**Example:** A task has `recommended_model: qwen-coder-mlx`, its convoy has `model_policy.roles: { role-implementer: claude-planning }`, and the workspace route maps `role-implementer → qwen-coder`. The system picks **`qwen-coder-mlx`** (brief override wins).
|
||||
|
||||
### 2. Registry Components
|
||||
|
||||
| Component | Table | Purpose |
|
||||
|-----------|-------|---------|
|
||||
| **Model Provider** | `model_providers` | Concrete model config: API endpoint, auth, params |
|
||||
| **Role Route** | `role_model_routes` | Which alias serves which role |
|
||||
| **Convoy Override** | `.convoys/*.md` frontmatter | Per-convoy role mapping |
|
||||
| **Brief Override** | `.convoys/<slug>/brief-*.md` frontmatter | Per-task specific model |
|
||||
|
||||
### 3. OmniRoute Integration
|
||||
|
||||
**OmniRoute profiles** map to Echodo roles:
|
||||
|
||||
| OmniRoute Profile | Echodo Role | Use Case |
|
||||
|-------------------|-------------|----------|
|
||||
| `auto/reasoning` | `role-conductor`, `role-architect` | Planning, classification, design |
|
||||
| `auto/reasoning-pro` | `role-architect` (escalation), `gate-copilot` | Complex reasoning, escalation path |
|
||||
| `auto/coding` | `role-implementer`, `role-doc-writer` | Code/docs implementation |
|
||||
| `auto/coding-pro` | `role-reviewer`, `role-implementer` (escalation) | Code review, complex refactoring |
|
||||
| `auto` | `chat-default`, fallback | General purpose, fastest |
|
||||
|
||||
**How it works:**
|
||||
1. Echodo resolves alias via `resolveModel(workspace, roleId)`
|
||||
2. Returns `ResolvedModel` with provider type, model ID, base URL, env var name
|
||||
3. If provider is `omni_route` type (or base URL points to OmniRoute), OmniRoute resolves the alias to a live endpoint
|
||||
4. Worker uses resolved endpoint for inference
|
||||
|
||||
### 4. The `ResolvedModel` Contract
|
||||
|
||||
```typescript
|
||||
// Runtime structure (not persisted)
|
||||
type ResolvedModel = {
|
||||
alias: string; // "qwen-coder-mlx"
|
||||
providerType: ProviderType; // "openai_compatible"
|
||||
modelId: string; // "mlx-community/Qwen2.5-Coder-32B-Instruct-4bit"
|
||||
baseUrl?: string; // "http://192.168.1.42:8000/v1"
|
||||
apiKey: string; // From process.env[apiKeyEnvVar]
|
||||
extraHeaders?: Record<string, string>;
|
||||
defaultParams?: { temperature?: number; maxTokens?: number };
|
||||
tier?: "premium" | "fast" | "local" | "auto";
|
||||
};
|
||||
```
|
||||
|
||||
**Key invariants:**
|
||||
- Never store API keys in DB → only `apiKeyEnvVar` names
|
||||
- Worker must have env var populated at runtime
|
||||
- `baseUrl` required for `openai_compatible` provider type
|
||||
- `tier` is advisory (cost/performance grouping)
|
||||
|
||||
## Model Provider Types
|
||||
|
||||
### Provider Type Matrix
|
||||
|
||||
| Type | Client | API Format | Example Alias | Use For |
|
||||
|------|--------|------------|---------------|---------|
|
||||
| `anthropic` | `@ai-sdk/anthropic` | Anthropic Messages API | `claude-planning` | Planning, reasoning, complex tasks |
|
||||
| `openai` | `@ai-sdk/openai` | OpenAI Chat Completion | `gpt-4o-mini` | General, chat |
|
||||
| `google` | `@ai-sdk/google` | Gemini REST API | `gemini-flash` | Multimodal, reasoning |
|
||||
| `openai_compatible` | `@ai-sdk/openai` + `baseURL` | OpenAI-compatible | `qwen-coder-mlx`, `nim-llama-70b` | Local inference, NIM |
|
||||
| `azure_openai` | `@ai-sdk/openai` + Azure config | Azure OpenAI | `azure-gpt4` | Enterprise Azure deployments |
|
||||
| `omni_route` | Custom client | OmniRoute API | `auto/coding` | Dynamic routing across 929 models |
|
||||
|
||||
### Local Inference Setup Examples
|
||||
|
||||
**Ollama (Mac):**
|
||||
```yaml
|
||||
alias: qwen-coder
|
||||
provider_type: openai_compatible
|
||||
model_id: qwen2.5-coder:32b
|
||||
base_url: http://192.168.1.42:11434/v1
|
||||
api_key_env_var: OLLAMA_API_KEY # placeholder, any non-empty string
|
||||
metadata: { tier: local, runtime: ollama }
|
||||
```
|
||||
|
||||
**MLX (Apple Silicon via mlx-openai-server):**
|
||||
```yaml
|
||||
alias: qwen-coder-mlx
|
||||
provider_type: openai_compatible
|
||||
model_id: mlx-community/Qwen2.5-Coder-32B-Instruct-4bit
|
||||
base_url: http://192.168.1.42:8000/v1
|
||||
api_key_env_var: MLX_API_KEY
|
||||
metadata: { tier: local, runtime: mlx }
|
||||
```
|
||||
|
||||
**Parallel implementers:** Use `mlx-openai-server` (port 8000) over `mlx_lm.server` (8080) for better batching when orchestrator fans out to multiple implementers.
|
||||
|
||||
**OmniRoute Cloud:**
|
||||
```yaml
|
||||
alias: auto/coding
|
||||
provider_type: omni_route
|
||||
model_id: auto/coding
|
||||
base_url: https://omniroute-cloud.example.com/v1
|
||||
api_key_env_var: OMNIROUTE_API_KEY
|
||||
metadata: { tier: auto }
|
||||
```
|
||||
|
||||
## How to Configure
|
||||
|
||||
### 1. Create Provider (Settings → AI → Providers)
|
||||
|
||||
```typescript
|
||||
// Example: Add local MLX provider
|
||||
{
|
||||
alias: "qwen-coder-mlx",
|
||||
display_name: "Qwen2.5 Coder MLX",
|
||||
provider_type: "openai_compatible",
|
||||
model_id: "mlx-community/Qwen2.5-Coder-32B-Instruct-4bit",
|
||||
base_url: "http://192.168.1.42:8000/v1",
|
||||
api_key_env_var: "MLX_API_KEY",
|
||||
metadata: { tier: "local", runtime: "mlx" }
|
||||
}
|
||||
```
|
||||
|
||||
**Test:** Click "Test" button in UI. It sends a small token request to verify connectivity.
|
||||
|
||||
### 2. Map Role to Alias (Settings → AI → Role Routing)
|
||||
|
||||
```typescript
|
||||
// Example: Route implementers to local MLX
|
||||
{
|
||||
role_id: "role-implementer",
|
||||
provider_alias_id: "qwen-coder-mlx-id"
|
||||
}
|
||||
```
|
||||
|
||||
**Supported role IDs:**
|
||||
- `role-conductor`, `role-architect`, `role-implementer`, `role-reviewer`
|
||||
- `role-ia-architect`, `role-ux-reviewer`, `role-design-system-auditor`, `role-a11y-auditor`, `role-doc-writer`
|
||||
- `gate-copilot`, `chat-default`
|
||||
- Custom roles (string IDs, no migration needed)
|
||||
|
||||
### 3. Convoy Override (`.convoys/*.md`)
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: feature-x
|
||||
model_policy:
|
||||
roles:
|
||||
role-conductor: claude-planning
|
||||
role-architect: claude-planning
|
||||
role-implementer: qwen-coder-mlx
|
||||
role-reviewer: qwen-coder-mlx
|
||||
escalate_to: claude-planning
|
||||
---
|
||||
```
|
||||
|
||||
### 4. Brief Override (`.convoys/<slug>/brief-*.md`)
|
||||
|
||||
```yaml
|
||||
---
|
||||
brief: implement-feature
|
||||
recommended_model: qwen-coder-mlx
|
||||
model_tier: local
|
||||
---
|
||||
```
|
||||
|
||||
## Resolution API
|
||||
|
||||
### `resolveModel(workspaceId, roleId, context?)`
|
||||
|
||||
```typescript
|
||||
// In packages/shared/src/orchestrator/resolve.ts
|
||||
function resolveModel(input: {
|
||||
workspaceId: string;
|
||||
roleId: string;
|
||||
briefContext?: { recommended_model?: string };
|
||||
convoyContext?: { model_policy?: Record<string, string> };
|
||||
}): ResolvedModel
|
||||
```
|
||||
|
||||
**Implementation steps:**
|
||||
1. If `briefContext?.recommended_model` exists, use that alias
|
||||
2. Else if `convoyContext?.model_policy?.[roleId]` exists, use that alias
|
||||
3. Else lookup `role_model_routes` row for `(workspaceId, roleId)`
|
||||
4. Else use default mapping (pipeline template)
|
||||
5. Load provider config for alias from `model_providers`
|
||||
6. Validate provider is enabled
|
||||
7. Read API key from env var
|
||||
8. Return `ResolvedModel`
|
||||
|
||||
### Orchestrator Usage
|
||||
|
||||
```typescript
|
||||
// When dispatching a job
|
||||
const resolved = await resolveModel({
|
||||
workspaceId: convoy.workspace_id,
|
||||
roleId: brief.role_id,
|
||||
briefContext: { recommended_model: brief.recommended_model },
|
||||
convoyContext: { model_policy: convoy.model_policy }
|
||||
});
|
||||
|
||||
// Job payload includes model spec
|
||||
await enqueueJob({
|
||||
roleId: brief.role_id,
|
||||
briefSlug: brief.slug,
|
||||
modelSpec: resolved,
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
## Escalation Logic
|
||||
|
||||
When `model_policy.escalate_to` is set or workspace policy defines escalation:
|
||||
|
||||
```typescript
|
||||
// After N implementer failures (default 2)
|
||||
if (attempts >= 2 && convoy.model_policy?.escalate_to) {
|
||||
const escalated = await resolveModel({
|
||||
workspaceId: convoy.workspace_id,
|
||||
roleId: roleId,
|
||||
convoyContext: { model_policy: { [roleId]: convoy.model_policy.escalate_to } }
|
||||
});
|
||||
// Re-dispatch with escalated model
|
||||
}
|
||||
```
|
||||
|
||||
## Logging Model Metadata
|
||||
|
||||
**`agent_runs` columns:**
|
||||
|
||||
| Column | Source | Example |
|
||||
|--------|--------|---------|
|
||||
| `model_alias` | `ResolvedModel.alias` | `qwen-coder-mlx` |
|
||||
| `provider_type` | `ResolvedModel.providerType` | `openai_compatible` |
|
||||
| `model_id` | `ResolvedModel.modelId` | `mlx-community/Qwen2.5-Coder-32B-Instruct-4bit` |
|
||||
| `model_tier` | `ResolvedModel.tier` | `local` |
|
||||
|
||||
**MCP `complete_task` fields:**
|
||||
|
||||
```typescript
|
||||
await complete_task({
|
||||
// ...
|
||||
modelAlias: "qwen-coder-mlx",
|
||||
modelTier: "local",
|
||||
tokensTotal: tokens.input + tokens.output,
|
||||
});
|
||||
```
|
||||
|
||||
**`convoy_events` role_complete:**
|
||||
|
||||
```typescript
|
||||
await log_convoy_event({
|
||||
// ...
|
||||
metadata: {
|
||||
model: "qwen-coder-mlx",
|
||||
tier: "local",
|
||||
tokens_in: 12345,
|
||||
tokens_out: 6789,
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
## Default Mappings (Pipeline Template)
|
||||
|
||||
New workspaces get these seed routes:
|
||||
|
||||
| Role ID | Default Alias | Tier | Purpose |
|
||||
|---------|---------------|------|---------|
|
||||
| `role-conductor` | `claude-planning` | premium | Kickoff, classification |
|
||||
| `role-architect` | `claude-planning` | premium | Design, brief decomposition |
|
||||
| `role-ia-architect` | `claude-planning` | premium | Information architecture |
|
||||
| `role-ux-reviewer` | `qwen-coder` | fast | UI/UX review |
|
||||
| `role-implementer` | `qwen-coder` | fast | Code implementation |
|
||||
| `role-reviewer` | `qwen-coder` | fast | Code review |
|
||||
| `role-design-system-auditor` | `qwen-coder` | fast | Design system audit |
|
||||
| `role-a11y-auditor` | `qwen-coder` | fast | Accessibility audit |
|
||||
| `role-doc-writer` | `qwen-coder` | fast | Documentation writing |
|
||||
| `gate-copilot` | `claude-planning` | premium | Gate decision support |
|
||||
| `chat-default` | workspace choice | — | Web chat default |
|
||||
|
||||
Operators can override these defaults in Settings.
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Setting Up Local MLX + OmniRoute Hybrid
|
||||
|
||||
**Goal:** Use local MLX for coding, OmniRoute for reasoning.
|
||||
|
||||
1. **Create local MLX provider:** `qwen-coder-mlx` (port 8000)
|
||||
2. **Create OmniRoute provider:** `auto/reasoning` (cloud endpoint)
|
||||
3. **Route roles:**
|
||||
- `role-implementer` → `qwen-coder-mlx`
|
||||
- `role-conductor` → `auto/reasoning`
|
||||
- `role-architect` → `auto/reasoning-pro`
|
||||
4. **Convoy example:**
|
||||
```yaml
|
||||
model_policy:
|
||||
roles:
|
||||
role-conductor: auto/reasoning
|
||||
role-implementer: qwen-coder-mlx
|
||||
```
|
||||
|
||||
### Debugging Resolution Failures
|
||||
|
||||
**Symptoms:** Job fails with "Cannot resolve model for role-implementer"
|
||||
|
||||
**Debug steps:**
|
||||
1. Check brief frontmatter: `recommended_model` present?
|
||||
2. Check convoy frontmatter: `model_policy.roles[role-implementer]` present?
|
||||
3. Query DB: `SELECT * FROM role_model_routes WHERE workspace_id = ? AND role_id = 'role-implementer'`
|
||||
4. Check provider: Is alias enabled in `model_providers`?
|
||||
5. Test provider: Run `testProvider` MCP tool for the alias
|
||||
6. Check env var: Is `api_key_env_var` populated in worker process?
|
||||
|
||||
### Migration from Cursor Model Slugs
|
||||
|
||||
Legacy convoys may have Cursor slugs (`claude-4.6-opus-high-thinking`). Add legacy alias mapping in workspace metadata:
|
||||
|
||||
```json
|
||||
{
|
||||
"legacy_model_aliases": {
|
||||
"claude-4.6-opus-high-thinking": "claude-planning",
|
||||
"composer-2.5-fast": "qwen-coder"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
During resolution, check `legacy_model_aliases[alias]` before failing.
|
||||
|
||||
### OmniRoute as the Sole Provider
|
||||
|
||||
If you want all routing via OmniRoute:
|
||||
|
||||
1. **Create OmniRoute providers** for all 5 profiles
|
||||
- `auto`, `auto/coding`, `auto/coding-pro`, `auto/reasoning`, `auto/reasoning-pro`
|
||||
2. **Route roles** accordingly
|
||||
3. **All inference** goes through OmniRoute cloud endpoint
|
||||
4. **Advantages:** Dynamic model selection, load balancing, fallbacks
|
||||
5. **Disadvantages:** Network dependency, cost (vs local free)
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Workspace with Local + Cloud Mix
|
||||
|
||||
**Providers:**
|
||||
```yaml
|
||||
# Local MLX (free, fast for coding)
|
||||
- alias: qwen-coder-mlx
|
||||
provider_type: openai_compatible
|
||||
base_url: http://mac:8000/v1
|
||||
tier: local
|
||||
|
||||
# Cloud Claude (costly, good for planning)
|
||||
- alias: claude-planning
|
||||
provider_type: anthropic
|
||||
tier: premium
|
||||
|
||||
# OmniRoute fallback
|
||||
- alias: auto/reasoning
|
||||
provider_type: omni_route
|
||||
tier: auto
|
||||
```
|
||||
|
||||
**Role routes:**
|
||||
- `role-conductor` → `claude-planning`
|
||||
- `role-implementer` → `qwen-coder-mlx`
|
||||
- `role-reviewer` → `qwen-coder-mlx`
|
||||
- `gate-copilot` → `auto/reasoning` (fallback when Claude unavailable)
|
||||
|
||||
**Convoy policy:**
|
||||
```yaml
|
||||
model_policy:
|
||||
roles:
|
||||
role-conductor: claude-planning
|
||||
role-implementer: qwen-coder-mlx
|
||||
escalate_to: auto/reasoning
|
||||
```
|
||||
|
||||
### Example 2: Brief Override for Complex Task
|
||||
|
||||
**Brief:** `.convoys/optimize-db/brief-5-complex-migration.md`
|
||||
|
||||
```yaml
|
||||
---
|
||||
brief: complex-db-migration
|
||||
recommended_model: auto/reasoning-pro
|
||||
model_tier: premium
|
||||
---
|
||||
|
||||
# Complex schema migration
|
||||
Needs reasoning about foreign keys and data integrity.
|
||||
```
|
||||
|
||||
The `recommended_model: auto/reasoning-pro` overrides the convoy's `model_policy.roles[role-implementer]`.
|
||||
|
||||
### Example 3: Testing Provider Connectivity
|
||||
|
||||
```bash
|
||||
# Via MCP tool
|
||||
testProvider({ alias: "qwen-coder-mlx" })
|
||||
|
||||
# Response
|
||||
{
|
||||
"success": true,
|
||||
"latency_ms": 245,
|
||||
"model": "mlx-community/Qwen2.5-Coder-32B-Instruct-4bit"
|
||||
}
|
||||
|
||||
# Error case
|
||||
{
|
||||
"success": false,
|
||||
"error": "Connection refused",
|
||||
"detail": "Check base_url and that mlx-openai-server is running on port 8000"
|
||||
}
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- **ADR:** `docs/adr-model-routing.md` (full specification)
|
||||
- **Schema:** `packages/database/src/schema/model_providers.ts`, `role_model_routes.ts`
|
||||
- **Resolver:** `packages/shared/src/orchestrator/resolve.ts` (or equivalent)
|
||||
- **OmniRoute docs:** `.convoys/model-routing-registry.md` if exists
|
||||
- **Local inference:** `docs/adr-model-routing.md#local-mlx-on-apple-silicon-mac`
|
||||
- **Worker integration:** `docs/worker.md#model-resolution`
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `backlog-sync` — brief frontmatter with `recommended_model`
|
||||
- `convoy-lifecycle` — convoy frontmatter with `model_policy`
|
||||
- `openhands-worker` — uses `ResolvedModel` for inference
|
||||
- `omniroute-setup` — operational guide for OmniRoute integration
|
||||
343
.agents/skills/tdd-red-green/SKILL.md
Normal file
343
.agents/skills/tdd-red-green/SKILL.md
Normal file
|
|
@ -0,0 +1,343 @@
|
|||
---
|
||||
name: tdd-red-green
|
||||
description: Test-Driven Development loop: write a failing test (Red), make it pass with minimal code (Green), then refactor (Refactor). Write tests before implementation, use seams for isolation, and avoid common TDD anti-patterns. Use when implementing features, fixing bugs, or adding tests to existing code.
|
||||
---
|
||||
|
||||
# TDD: Red-Green-Refactor
|
||||
|
||||
A practical test-driven development loop adapted for Echodo's codebase. Write the test first, make it pass with minimal code, then refactor. This skill teaches you the TDD cycle, how to write effective tests, and common pitfalls to avoid.
|
||||
|
||||
## When to Activate
|
||||
|
||||
Use this skill when:
|
||||
- Implementing a new feature (add tests first)
|
||||
- Fixing a bug (write a test that reproduces it, then fix)
|
||||
- Adding coverage to existing code
|
||||
- Following Echodo's TDD convention in `role-implementer` briefs
|
||||
|
||||
## The Core Loop
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ 1. Write failing test (Red) │
|
||||
│ → Test describes expected behavior │
|
||||
│ → Run test → fails (expected) │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 2. Write minimal code to pass (Green) │
|
||||
│ → Only enough to make test pass │
|
||||
│ → No "preemptive" abstractions │
|
||||
│ → Run test → passes │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ 3. Refactor (Refactor) │
|
||||
│ → Clean up without changing behavior │
|
||||
│ → Keep tests passing │
|
||||
│ → Run full test suite │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Step 1: Write Failing Test (Red)
|
||||
|
||||
### What Makes a Good Test?
|
||||
|
||||
1. **Describes behavior, not implementation** — "should create user with valid email" not "should call User.create"
|
||||
2. **Single assertion per test** (or very few) — easier to diagnose failures
|
||||
3. **Test name explains the scenario** — `describe('when email is invalid', () => { it('should return 400') })`
|
||||
4. **Arrange-Act-Assert structure** — clear setup, action, and expectation
|
||||
|
||||
### Test Location
|
||||
|
||||
Follow Echodo conventions:
|
||||
- Tests next to source: `src/index.ts` → `src/index.test.ts`
|
||||
- Test file suffix: `.test.ts` (Vitest)
|
||||
- Use existing test utilities in `packages/shared/test/` if available
|
||||
|
||||
### Example Red Test
|
||||
|
||||
```typescript
|
||||
// apps/web/server/routers/auth.test.ts
|
||||
describe('POST /api/auth/register', () => {
|
||||
it('should create a user with valid email and password', async () => {
|
||||
// Arrange
|
||||
const payload = {
|
||||
email: 'test@example.com',
|
||||
password: 'securepassword123'
|
||||
};
|
||||
|
||||
// Act
|
||||
const response = await fetch('/api/auth/register', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
|
||||
// Assert
|
||||
expect(response.status).toBe(201);
|
||||
const body = await response.json();
|
||||
expect(body.user.email).toBe('test@example.com');
|
||||
});
|
||||
|
||||
it('should return 400 when email is invalid', async () => {
|
||||
// Arrange
|
||||
const payload = {
|
||||
email: 'not-an-email',
|
||||
password: 'securepassword123'
|
||||
};
|
||||
|
||||
// Act
|
||||
const response = await fetch('/api/auth/register', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
|
||||
// Assert
|
||||
expect(response.status).toBe(400);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
**Run test → fails** (because route doesn't exist yet)
|
||||
|
||||
### When Bug-Fixing: Start with a Failing Test
|
||||
|
||||
Instead of:
|
||||
1. Reading the bug
|
||||
2. Fixing the code
|
||||
3. Running full test suite
|
||||
|
||||
Do this:
|
||||
1. Write a test that **reproduces the bug**
|
||||
2. Run test → fails (proves bug exists)
|
||||
3. Fix code
|
||||
4. Run test → passes (proves bug fixed)
|
||||
5. Run full suite
|
||||
|
||||
## Step 2: Write Minimal Code (Green)
|
||||
|
||||
### Key Principles
|
||||
|
||||
1. **Minimal implementation** — only enough to pass the test
|
||||
2. **No premature abstraction** — don't build "flexible" systems "just in case"
|
||||
3. **YAGNI** — You Aren't Gonna Need It
|
||||
4. **Trust the test** — if the test is right, the code should make it pass
|
||||
|
||||
### Example Green Implementation
|
||||
|
||||
```typescript
|
||||
// Minimal route to make test pass
|
||||
// apps/web/app/api/auth/register/route.ts
|
||||
import { NextResponse } from 'next/server';
|
||||
import { z } from 'zod';
|
||||
|
||||
const registerSchema = z.object({
|
||||
email: z.string().email(),
|
||||
password: z.string().min(8)
|
||||
});
|
||||
|
||||
export async function POST(request: Request) {
|
||||
const body = await request.json();
|
||||
|
||||
// Basic validation (the test checks this)
|
||||
const result = registerSchema.safeParse(body);
|
||||
if (!result.success) {
|
||||
return NextResponse.json({ error: 'Invalid input' }, { status: 400 });
|
||||
}
|
||||
|
||||
// Minimal DB insert (simplified for example)
|
||||
const user = await createUser(result.data.email, result.data.password);
|
||||
|
||||
return NextResponse.json({ user }, { status: 201 });
|
||||
}
|
||||
|
||||
// Placeholder - in real code, call actual DB
|
||||
async function createUser(email: string, password: string) {
|
||||
return { id: '1', email };
|
||||
}
|
||||
```
|
||||
|
||||
**Run test → passes**
|
||||
|
||||
## Step 3: Refactor
|
||||
|
||||
### When to Refactor
|
||||
|
||||
- Code is duplicated
|
||||
- Names are unclear
|
||||
- Function does too much
|
||||
- Test is fragile or implementation-specific
|
||||
- You need to add a new test that requires a small change
|
||||
|
||||
### While Refactoring
|
||||
|
||||
1. **Keep tests passing** — run after every change
|
||||
2. **Small steps** — refactor one thing at a time
|
||||
3. **No new functionality** — only clean up existing code
|
||||
|
||||
### Example Refactor
|
||||
|
||||
Before:
|
||||
```typescript
|
||||
// auth.ts - duplicated validation
|
||||
function validateEmail(email: string) {
|
||||
return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email);
|
||||
}
|
||||
|
||||
// Different file, same validation
|
||||
function isValidEmailAddress(email: string) {
|
||||
return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email);
|
||||
}
|
||||
```
|
||||
|
||||
After (extract to shared):
|
||||
```typescript
|
||||
// lib/validation.ts
|
||||
export const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
|
||||
export function isValidEmail(email: string): boolean {
|
||||
return emailRegex.test(email);
|
||||
}
|
||||
```
|
||||
|
||||
## Seams: Testing Without Dependencies
|
||||
|
||||
A **seam** is a place where you can alter behavior without changing the code being tested.
|
||||
|
||||
### Types of Seams
|
||||
|
||||
| Seam | How to Use | Example |
|
||||
|------|------------|---------|
|
||||
| **Dependency Injection** | Pass dependencies as params | `function process(db)` → `process(mockDb)` |
|
||||
| **Interface/Abstract class** | Swap implementations | `Database` abstract → `MockDatabase` |
|
||||
| **Factory functions** | Return different instances | `createClient()` → `createMockClient()` |
|
||||
| **Environment variables** | Control behavior | `if (process.env.NODE_ENV === 'test')` |
|
||||
| **Configuration objects** | Pass config at runtime | `new Service({ retries: 3 })` |
|
||||
|
||||
### Example: Seams in Echodo
|
||||
|
||||
```typescript
|
||||
// Without seam (hard to test)
|
||||
async function createUser(email: string) {
|
||||
const db = new PostgresDatabase(); // hard-coded
|
||||
return db.users.insert({ email });
|
||||
}
|
||||
|
||||
// With seam (testable)
|
||||
async function createUser(email: string, db: Database = defaultDb) {
|
||||
return db.users.insert({ email });
|
||||
}
|
||||
|
||||
// In test
|
||||
const mockDb = { users: { insert: vi.fn() } };
|
||||
await createUser('test@example.com', mockDb);
|
||||
expect(mockDb.users.insert).toHaveBeenCalledWith({ email: 'test@example.com' });
|
||||
```
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
### 1. Test After Implementation
|
||||
|
||||
❌ Write code → write tests
|
||||
✅ Write failing test → write code → pass → refactor
|
||||
|
||||
### 2. Implementation-Specific Tests
|
||||
|
||||
❌ `expect(service.createUser).toHaveBeenCalledWith(data)`
|
||||
✅ `expect(response.status).toBe(201)` (behavior, not implementation)
|
||||
|
||||
### 3. Brittle Tests
|
||||
|
||||
❌ Test checks exact string: `expect(body).toBe('{"id":"1"}')`
|
||||
✅ Test checks structure: `expect(body.user.id).toBeDefined()`
|
||||
|
||||
### 4. Shared State Between Tests
|
||||
|
||||
❌ Global variable modified in one test, relied on by another
|
||||
✅ Each test sets up its own state
|
||||
|
||||
### 5. Testing Too Much
|
||||
|
||||
❌ One test checks validation, DB insert, email send, response format
|
||||
✅ Separate tests for each concern
|
||||
|
||||
### 6. No Assertions
|
||||
|
||||
❌ `await service.process(data)` — no expect
|
||||
✅ `expect(result).toEqual(expected)`
|
||||
|
||||
### 7. Commented-Out Tests
|
||||
|
||||
❌ `// skip this test for now`
|
||||
✅ Delete or fix the test
|
||||
|
||||
## Echodo Testing Conventions
|
||||
|
||||
- **Framework:** Vitest (see `packages/*/vitest.config.ts`)
|
||||
- **Location:** Same directory as source, `.test.ts` suffix
|
||||
- **Mocking:** Use `vi.fn()` from Vitest, mock via dependency injection
|
||||
- **Database tests:** Use test database or in-memory alternative
|
||||
- **Run tests:** `pnpm test` or `pnpm --filter @tasks/<package> test`
|
||||
|
||||
## Example: Bug Fix Flow
|
||||
|
||||
**Bug report:** "Users can register with empty password"
|
||||
|
||||
### Before (wrong approach)
|
||||
1. Read bug report
|
||||
2. Add validation to route
|
||||
3. Run full test suite
|
||||
4. Deploy
|
||||
|
||||
### TDD Approach (right way)
|
||||
|
||||
1. **Write failing test:**
|
||||
```typescript
|
||||
it('should reject password shorter than 8 characters', async () => {
|
||||
const response = await register({ email: 'test@example.com', password: 'short' });
|
||||
expect(response.status).toBe(400);
|
||||
});
|
||||
```
|
||||
|
||||
2. **Run test → fails** (shows bug exists)
|
||||
|
||||
3. **Fix code minimally:**
|
||||
```typescript
|
||||
const registerSchema = z.object({
|
||||
email: z.string().email(),
|
||||
password: z.string().min(8)
|
||||
});
|
||||
```
|
||||
|
||||
4. **Run test → passes** (proves fix works)
|
||||
|
||||
5. **Run full suite** (no regressions)
|
||||
|
||||
6. **Refactor** if needed
|
||||
|
||||
## Integration with Other Skills
|
||||
|
||||
- **code-review-two-axis:** Reviewer checks for TDD pattern in implementation
|
||||
- **backlog-sync:** When testing markdown import/export features
|
||||
- **convoy-lifecycle:** When testing convoy status transitions
|
||||
- **model-routing:** When testing model resolution logic
|
||||
|
||||
## Running Tests in Echodo
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pnpm test
|
||||
|
||||
# Run specific package
|
||||
pnpm --filter @tasks/web test
|
||||
pnpm --filter @tasks/shared test
|
||||
|
||||
# Watch mode
|
||||
pnpm --filter @tasks/web test:watch
|
||||
|
||||
# Coverage
|
||||
pnpm test --coverage
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- **Original skill:** https://github.com/mattpocock/skills/blob/main/skills/engineering/tdd/SKILL.md
|
||||
- **Kent Beck:** "Test-Driven Development: By Example"
|
||||
- **Martin Fowler:** "Refactoring" - Chapter on code smells
|
||||
- **Vitest docs:** https://vitest.dev/
|
||||
- **Echodo test config:** `packages/*/vitest.config.ts`
|
||||
|
|
@ -33,6 +33,21 @@ const envSchema = z.object({
|
|||
.string()
|
||||
.optional()
|
||||
.transform((value) => value?.trim() || "/tmp/echodo-worker-worktrees"),
|
||||
/** Session type: "minimal" (default, in-process) or "openhands" (agent-server). */
|
||||
WORKER_SESSION_TYPE: z
|
||||
.enum(["minimal", "openhands"])
|
||||
.optional()
|
||||
.default("minimal"),
|
||||
/** OpenHands agent-server URL (local: http://localhost:18000, Cloud: from conversation). */
|
||||
AGENT_SERVER_URL: z
|
||||
.string()
|
||||
.optional()
|
||||
.transform((value) => value?.trim() || "http://localhost:18000"),
|
||||
/** OpenHands session API key for agent-server auth. */
|
||||
SESSION_API_KEY: z
|
||||
.string()
|
||||
.optional()
|
||||
.transform((value) => value?.trim() || ""),
|
||||
});
|
||||
|
||||
export type WorkerEnv = z.infer<typeof envSchema>;
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import type {
|
|||
WorkerJobContext,
|
||||
} from "@tasks/shared/worker";
|
||||
import { createMinimalSession } from "../session/minimal-session.js";
|
||||
import { createOpenHandsSession, type OpenHandsConfig } from "../session/openhands-session.js";
|
||||
import { createRemoteAgUiExecutor } from "./remote-ag-ui.js";
|
||||
|
||||
export type ExecutorDeps = {
|
||||
|
|
@ -16,7 +17,7 @@ function buildMinimalExecutor(): WorkerExecutor {
|
|||
return {
|
||||
id: "minimal",
|
||||
harnessType: "minimal" as HarnessType,
|
||||
async run(ctx: WorkerJobContext, opts: WorkerExecutorOptions): Promise<{ summary: string; toolCalls: number; tokensTotal?: number }> {
|
||||
async run(ctx: WorkerJobContext, opts: WorkerExecutorOptions): Promise<{ summary: string; toolCalls: number; tokensTotal?: number; outcome?: "succeeded" | "failed" }> {
|
||||
const session = createMinimalSession(ctx, {
|
||||
resolvedModel: opts.resolvedModel as never,
|
||||
resolvedAlias: opts.resolvedAlias,
|
||||
|
|
@ -54,6 +55,33 @@ function buildRemoteAgUiExecutor(
|
|||
});
|
||||
}
|
||||
|
||||
function buildOpenHandsExecutor(config: OpenHandsConfig): WorkerExecutor {
|
||||
return {
|
||||
id: "openhands",
|
||||
harnessType: "minimal" as HarnessType,
|
||||
async run(ctx: WorkerJobContext, opts: WorkerExecutorOptions): Promise<{ summary: string; toolCalls: number; tokensTotal?: number; outcome?: "succeeded" | "failed" }> {
|
||||
const session = await createOpenHandsSession(ctx, {
|
||||
config,
|
||||
resolvedModel: opts.resolvedModel as never,
|
||||
roleId: ctx.roleId,
|
||||
briefId: ctx.briefId,
|
||||
briefTitle: opts.convoyTitle,
|
||||
workflowPrompt: opts.workflowPrompt,
|
||||
briefAcceptance: opts.briefAcceptance,
|
||||
briefFiles: opts.briefFiles,
|
||||
maxSteps: opts.maxSteps,
|
||||
});
|
||||
const convoyTitle = opts.convoyTitle ?? "unknown";
|
||||
const result = await session.run(opts.buildPrompt(ctx, convoyTitle));
|
||||
return {
|
||||
summary: result.summary,
|
||||
toolCalls: result.toolCalls,
|
||||
tokensTotal: result.tokensTotal,
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export type HarnessExecutorConfig = {
|
||||
harnessType: HarnessType;
|
||||
endpointUrl?: string | null;
|
||||
|
|
@ -73,3 +101,19 @@ export function getExecutor(
|
|||
default: return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create an executor for OpenHands agent-server sessions.
|
||||
* Falls back to minimal if env vars are missing.
|
||||
*/
|
||||
export function createOpenHandsExecutor(
|
||||
agentServerUrl: string,
|
||||
sessionApiKey: string,
|
||||
repoRoot: string,
|
||||
): WorkerExecutor | null {
|
||||
if (!agentServerUrl || !sessionApiKey) {
|
||||
console.warn("[worker] OpenHands executor unavailable: missing AGENT_SERVER_URL or SESSION_API_KEY");
|
||||
return null;
|
||||
}
|
||||
return buildOpenHandsExecutor({ agentServerUrl, sessionApiKey, repoRoot });
|
||||
}
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ import {
|
|||
} from "./git/worktree.js";
|
||||
import { openDraftPR } from "./git/pr.js";
|
||||
import { resolveHarnessForJob, getHarnessExecutor } from "./harness.js";
|
||||
import { getExecutor, type HarnessExecutorConfig } from "./executor/index.js";
|
||||
import { getExecutor, createOpenHandsExecutor, type HarnessExecutorConfig } from "./executor/index.js";
|
||||
import { resolveHarnessForRoleDb } from "./db/harness.js";
|
||||
import { resolveRepoRoot } from "./lib/repo-root.js";
|
||||
import {
|
||||
|
|
@ -256,12 +256,34 @@ export async function handleOrchestratorJob(
|
|||
: undefined;
|
||||
|
||||
// Resolve the executor from the registry based on the harness type
|
||||
const harnessExecutorConfig: HarnessExecutorConfig = {
|
||||
harnessType: harness.harnessType,
|
||||
endpointUrl: harness.endpointUrl,
|
||||
authEnvVar: harness.authEnvVar,
|
||||
};
|
||||
const executor = getExecutor(harnessExecutorConfig);
|
||||
// When WORKER_SESSION_TYPE=openhands, route to OpenHands agent-server executor
|
||||
let executor;
|
||||
if (env.WORKER_SESSION_TYPE === "openhands") {
|
||||
const ohExecutor = createOpenHandsExecutor(
|
||||
env.AGENT_SERVER_URL,
|
||||
env.SESSION_API_KEY,
|
||||
monorepoRoot,
|
||||
);
|
||||
if (ohExecutor) {
|
||||
executor = ohExecutor;
|
||||
console.info(`[worker] using OpenHands executor agent_server=${env.AGENT_SERVER_URL}`);
|
||||
} else {
|
||||
console.warn("[worker] OpenHands executor unavailable, falling back to minimal");
|
||||
const harnessExecutorConfig: HarnessExecutorConfig = {
|
||||
harnessType: harness.harnessType,
|
||||
endpointUrl: harness.endpointUrl,
|
||||
authEnvVar: harness.authEnvVar,
|
||||
};
|
||||
executor = getExecutor(harnessExecutorConfig);
|
||||
}
|
||||
} else {
|
||||
const harnessExecutorConfig: HarnessExecutorConfig = {
|
||||
harnessType: harness.harnessType,
|
||||
endpointUrl: harness.endpointUrl,
|
||||
authEnvVar: harness.authEnvVar,
|
||||
};
|
||||
executor = getExecutor(harnessExecutorConfig);
|
||||
}
|
||||
|
||||
if (!executor) {
|
||||
outcome = "failed";
|
||||
|
|
@ -288,7 +310,7 @@ export async function handleOrchestratorJob(
|
|||
};
|
||||
|
||||
try {
|
||||
const result = await executor.run(sessionContext, modelResult.model, sessionOptions);
|
||||
const result = await executor.run(sessionContext, sessionOptions as import("@tasks/shared/worker").WorkerExecutorOptions);
|
||||
summary = result.summary;
|
||||
tokensTotal = result.tokensTotal;
|
||||
console.info(
|
||||
|
|
|
|||
105
apps/worker/src/session/load-skills.ts
Normal file
105
apps/worker/src/session/load-skills.ts
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
import { readdir, readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
/**
|
||||
* Loaded skill from `.agents/skills/<name>/SKILL.md`.
|
||||
* Parsed frontmatter + body is returned for injection into session system prompt.
|
||||
*/
|
||||
export type LoadedSkill = {
|
||||
name: string;
|
||||
description: string;
|
||||
body: string;
|
||||
};
|
||||
|
||||
const SKILL_FRONTMATTER = /^---\r?\n([\s\S]*?)\r?\n---\r?\n?/;
|
||||
|
||||
function parseSkillFrontmatter(raw: string): { name: string; description: string; body: string } {
|
||||
const match = raw.match(SKILL_FRONTMATTER);
|
||||
let name = "unknown";
|
||||
let description = "";
|
||||
let body = raw;
|
||||
|
||||
if (match) {
|
||||
body = raw.slice(match[0].length);
|
||||
const fm = match[1];
|
||||
const nameMatch = fm.match(/^name:\s*(.+)$/m);
|
||||
const descMatch = fm.match(/^description:\s*(.+)$/m);
|
||||
if (nameMatch) name = nameMatch[1].trim();
|
||||
if (descMatch) description = descMatch[1].trim();
|
||||
}
|
||||
|
||||
return { name, description, body };
|
||||
}
|
||||
|
||||
/**
|
||||
* Load all SKILL.md files from `.agents/skills/<name>/SKILL.md` relative to repo root.
|
||||
* Returns an array of parsed skills suitable for system prompt injection.
|
||||
* Failures per skill are logged and skipped (partial loading is acceptable).
|
||||
*/
|
||||
export async function loadAgentSkills(
|
||||
repoRoot: string,
|
||||
filterNames?: readonly string[],
|
||||
): Promise<LoadedSkill[]> {
|
||||
const skillsDir = join(repoRoot, ".agents", "skills");
|
||||
const skills: LoadedSkill[] = [];
|
||||
|
||||
let entries: import("node:fs").Dirent[];
|
||||
try {
|
||||
entries = await readdir(skillsDir, { withFileTypes: true });
|
||||
} catch {
|
||||
// .agents/skills/ not found — gracefully return empty
|
||||
return [];
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory()) continue;
|
||||
|
||||
const skillName = entry.name;
|
||||
if (filterNames && !filterNames.includes(skillName)) continue;
|
||||
|
||||
const skillPath = join(skillsDir, skillName, "SKILL.md");
|
||||
try {
|
||||
const raw = await readFile(skillPath, "utf8");
|
||||
const parsed = parseSkillFrontmatter(raw);
|
||||
skills.push({
|
||||
name: parsed.name || skillName,
|
||||
description: parsed.description,
|
||||
body: parsed.body,
|
||||
});
|
||||
} catch (e) {
|
||||
console.warn(`[worker] failed to load skill ${skillName}:`, e);
|
||||
}
|
||||
}
|
||||
|
||||
return skills;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a compact skill summary for the system prompt.
|
||||
* Includes skill name + description only (not full body) for non-verbose mode.
|
||||
* Use `loadAgentSkills(repoRoot)` for full content when needed.
|
||||
*/
|
||||
export function buildSkillSummary(skills: LoadedSkill[]): string {
|
||||
if (skills.length === 0) return "";
|
||||
const lines = skills.map((s) => `- **${s.name}**: ${s.description}`);
|
||||
return `\n\n## Available skills\n${lines.join("\n")}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build full skill content for the system prompt.
|
||||
* Includes full body of each skill for deep context.
|
||||
*/
|
||||
export function buildSkillFullContext(skills: LoadedSkill[]): string {
|
||||
if (skills.length === 0) return "";
|
||||
const blocks = skills.map((s) => `### Skill: ${s.name}\n${s.body}`);
|
||||
return `\n\n## Repo skills\n${blocks.join("\n\n")}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a prompt suffix listing available skills by name only (lightweight).
|
||||
*/
|
||||
export function buildSkillList(skills: LoadedSkill[]): string {
|
||||
if (skills.length === 0) return "";
|
||||
const names = skills.map((s) => s.name).join(", ");
|
||||
return `\n\nSkills available: ${names}`;
|
||||
}
|
||||
|
|
@ -19,6 +19,7 @@ import {
|
|||
runCommandTool,
|
||||
writeFileTool,
|
||||
} from "../tools/runner.js";
|
||||
import { loadAgentSkills, buildSkillFullContext } from "./load-skills.js";
|
||||
|
||||
export type CheckpointSink = (input: {
|
||||
checkpointSeq: number;
|
||||
|
|
@ -134,6 +135,18 @@ export function createMinimalSession(
|
|||
const checkpointEvery = Math.max(1, options.checkpointEveryNSteps ?? 1);
|
||||
const remainingSteps = Math.max(1, options.maxSteps - resumeHistory.length);
|
||||
|
||||
// Load skills from .agents/skills/ and inject into system prompt
|
||||
let skillContext = "";
|
||||
try {
|
||||
const skills = await loadAgentSkills(context.repoRoot);
|
||||
skillContext = buildSkillFullContext(skills);
|
||||
if (skills.length > 0) {
|
||||
console.info(`[worker] loaded ${skills.length} skills from .agents/skills/`);
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn("[worker] skill loading failed, continuing without skills:", e);
|
||||
}
|
||||
|
||||
let systemBase = [
|
||||
rolePrompt.trim(),
|
||||
options.workflowPrompt ? `\n\n## Workflow\n${options.workflowPrompt.trim()}` : "",
|
||||
|
|
@ -143,6 +156,7 @@ export function createMinimalSession(
|
|||
options.briefAcceptance
|
||||
? `\n\n## Acceptance\n${options.briefAcceptance.trim()}`
|
||||
: "",
|
||||
skillContext,
|
||||
"\n\nUse tools to inspect and modify the repo. Finish with a concise summary of work done.",
|
||||
buildResumeAppendix(resumeHistory),
|
||||
].join("");
|
||||
|
|
|
|||
283
apps/worker/src/session/openhands-session.ts
Normal file
283
apps/worker/src/session/openhands-session.ts
Normal file
|
|
@ -0,0 +1,283 @@
|
|||
import type { ResolvedModel } from "@tasks/shared/model-routing";
|
||||
import type {
|
||||
WorkerJobContext,
|
||||
WorkerSession,
|
||||
} from "@tasks/shared/worker";
|
||||
|
||||
import { loadAgentSkills, buildSkillFullContext } from "./load-skills.js";
|
||||
|
||||
/**
|
||||
* Configuration for the OpenHands agent server connection.
|
||||
* For local Agent Canvas: AGENT_SERVER_URL + SESSION_API_KEY (ambient).
|
||||
* For Cloud: OPENHANDS_CLOUD_API_KEY + agent_server_url from conversation.
|
||||
*/
|
||||
export type OpenHandsConfig = {
|
||||
agentServerUrl: string;
|
||||
sessionApiKey: string;
|
||||
/** Repo root to scan for .agents/skills/ */
|
||||
repoRoot: string;
|
||||
};
|
||||
|
||||
export type OpenHandsSessionOptions = {
|
||||
config: OpenHandsConfig;
|
||||
resolvedModel: ResolvedModel;
|
||||
roleId: string;
|
||||
briefId?: string;
|
||||
briefTitle?: string;
|
||||
workflowPrompt?: string;
|
||||
briefAcceptance?: string;
|
||||
briefFiles?: string[];
|
||||
maxSteps?: number;
|
||||
};
|
||||
|
||||
const POLL_INTERVAL_MS = 2000;
|
||||
const MAX_POLL_ATTEMPTS = 300; // 10 minutes at 2s interval
|
||||
|
||||
/**
|
||||
* Create an OpenHands session that runs an agent via the agent-server REST API.
|
||||
* The agent runs inside an OpenHands sandbox with full tool access (terminal, file_editor, etc.).
|
||||
*/
|
||||
export async function createOpenHandsSession(
|
||||
context: WorkerJobContext,
|
||||
options: OpenHandsSessionOptions,
|
||||
): Promise<WorkerSession> {
|
||||
const { config, resolvedModel, roleId, briefId, briefTitle, workflowPrompt, briefAcceptance, briefFiles } = options;
|
||||
const repoRoot = config.repoRoot;
|
||||
const apiBase = config.agentServerUrl.replace(/\/$/, "");
|
||||
const headers = {
|
||||
"X-Session-API-Key": config.sessionApiKey,
|
||||
"Content-Type": "application/json",
|
||||
} as const;
|
||||
|
||||
// Load skills from .agents/skills/
|
||||
let skillContext = "";
|
||||
try {
|
||||
const skills = await loadAgentSkills(repoRoot);
|
||||
skillContext = buildSkillFullContext(skills);
|
||||
} catch (e) {
|
||||
console.warn("[openhands] skill loading failed, continuing without skills:", e);
|
||||
}
|
||||
|
||||
// Build the system prompt that the agent will see
|
||||
const systemParts: string[] = [
|
||||
`You are an AI agent executing a brief for convoy "${context.convoyId}".`,
|
||||
`Role: ${roleId}`,
|
||||
briefId ? `Brief ID: ${briefId}` : "",
|
||||
briefTitle ? `Brief: ${briefTitle}` : "",
|
||||
workflowPrompt ? `\n## Task\n${workflowPrompt}` : "",
|
||||
briefAcceptance ? `\n## Acceptance criteria\n${briefAcceptance}` : "",
|
||||
briefFiles?.length ? `\n## Files to examine\n${briefFiles.map((f) => `- ${f}`).join("\n")}` : "",
|
||||
skillContext || "",
|
||||
"\n## Instructions\nUse the available tools to inspect and modify the repo. Run lint/type-check when appropriate. Finish with a concise summary.",
|
||||
].filter(Boolean);
|
||||
|
||||
const systemPrompt = systemParts.join("\n");
|
||||
|
||||
/**
|
||||
* Start a conversation on the OpenHands agent server.
|
||||
* Uses the local backend API (POST /api/conversations).
|
||||
*/
|
||||
async function startConversation(): Promise<{ conversationId: string }> {
|
||||
const body = {
|
||||
agent: {
|
||||
kind: "Agent",
|
||||
llm: {
|
||||
model: resolvedModel.modelId,
|
||||
api_key: resolvedModel.apiKey || "placeholder",
|
||||
base_url: resolvedModel.baseUrl,
|
||||
},
|
||||
tools: [
|
||||
{ name: "terminal" },
|
||||
{ name: "file_editor" },
|
||||
{ name: "task_tracker" },
|
||||
],
|
||||
},
|
||||
workspace: {
|
||||
kind: "LocalWorkspace",
|
||||
working_dir: repoRoot,
|
||||
},
|
||||
initial_message: {
|
||||
content: [{ text: systemPrompt }],
|
||||
run: true,
|
||||
},
|
||||
};
|
||||
|
||||
console.info(`[openhands] starting conversation roleId=${roleId} model=${resolvedModel.alias}`);
|
||||
|
||||
const res = await fetch(`${apiBase}/api/conversations`, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
|
||||
if (!res.ok) {
|
||||
const text = await res.text();
|
||||
throw new Error(`OpenHands start failed (${res.status}): ${text}`);
|
||||
}
|
||||
|
||||
const data = await res.json() as { id: string };
|
||||
return { conversationId: data.id };
|
||||
}
|
||||
|
||||
/**
|
||||
* Poll conversation status until it reaches a terminal state.
|
||||
*/
|
||||
async function pollUntilComplete(conversationId: string): Promise<{
|
||||
status: "completed" | "failed" | "timeout";
|
||||
summary: string;
|
||||
toolCalls: number;
|
||||
tokensTotal?: number;
|
||||
}> {
|
||||
for (let i = 0; i < MAX_POLL_ATTEMPTS; i++) {
|
||||
await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS));
|
||||
|
||||
const res = await fetch(
|
||||
`${apiBase}/api/conversations/${conversationId}`,
|
||||
{ headers },
|
||||
);
|
||||
if (!res.ok) {
|
||||
console.warn(`[openhands] poll failed (${res.status}), retrying...`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const conv = await res.json() as {
|
||||
execution_status?: string;
|
||||
sandbox_status?: string;
|
||||
};
|
||||
|
||||
const execStatus = conv.execution_status ?? "";
|
||||
const sandboxStatus = conv.sandbox_status ?? "";
|
||||
|
||||
if (execStatus === "completed" || execStatus === "finished") {
|
||||
return await fetchResult(conversationId, "completed");
|
||||
}
|
||||
if (execStatus === "failed" || sandboxStatus === "error") {
|
||||
return await fetchResult(conversationId, "failed");
|
||||
}
|
||||
|
||||
if (i % 10 === 0) {
|
||||
console.info(`[openhands] poll ${i} exec=${execStatus} sandbox=${sandboxStatus}`);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
status: "timeout",
|
||||
summary: "OpenHands conversation timed out after 10 minutes",
|
||||
toolCalls: 0,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch the final result from a completed/failed conversation.
|
||||
*/
|
||||
async function fetchResult(
|
||||
conversationId: string,
|
||||
fallbackStatus: "completed" | "failed",
|
||||
): Promise<{
|
||||
status: "completed" | "failed";
|
||||
summary: string;
|
||||
toolCalls: number;
|
||||
tokensTotal?: number;
|
||||
}> {
|
||||
try {
|
||||
const res = await fetch(
|
||||
`${apiBase}/api/conversations/${conversationId}/events/search?limit=500&sort_order=TIMESTAMP_DESC`,
|
||||
{ headers },
|
||||
);
|
||||
if (!res.ok) {
|
||||
return { status: fallbackStatus, summary: `Failed to fetch events (${res.status})`, toolCalls: 0 };
|
||||
}
|
||||
|
||||
const data = await res.json() as { items: Array<{
|
||||
kind: string;
|
||||
source: string;
|
||||
message?: string;
|
||||
tool_name?: string;
|
||||
tool_call_id?: string;
|
||||
action?: { command?: string; path?: string };
|
||||
observation?: { stdout?: string; exit_code?: number };
|
||||
}> };
|
||||
|
||||
const events = data.items ?? [];
|
||||
let toolCalls = 0;
|
||||
const assistantMessages: string[] = [];
|
||||
let tokensTotal: number | undefined;
|
||||
|
||||
for (const event of events) {
|
||||
if (event.kind === "ActionEvent" && event.source === "agent") {
|
||||
toolCalls++;
|
||||
}
|
||||
if (event.kind === "MessageEvent" && event.source === "assistant") {
|
||||
if (event.message) assistantMessages.push(event.message);
|
||||
}
|
||||
}
|
||||
|
||||
const summary = assistantMessages[0] ?? "OpenHands session completed with no summary.";
|
||||
|
||||
return {
|
||||
status: fallbackStatus,
|
||||
summary,
|
||||
toolCalls,
|
||||
tokensTotal,
|
||||
};
|
||||
} catch (e) {
|
||||
return {
|
||||
status: fallbackStatus,
|
||||
summary: `Failed to fetch result: ${e instanceof Error ? e.message : String(e)}`,
|
||||
toolCalls: 0,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Send a follow-up message to a running conversation.
|
||||
*/
|
||||
async function sendMessage(conversationId: string, text: string): Promise<void> {
|
||||
const res = await fetch(
|
||||
`${apiBase}/api/conversations/${conversationId}/send-message`,
|
||||
{
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify({ content: [{ text }] }),
|
||||
},
|
||||
);
|
||||
if (!res.ok) {
|
||||
const body = await res.text();
|
||||
console.warn(`[openhands] send-message failed (${res.status}): ${body}`);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
context,
|
||||
async run(prompt: string) {
|
||||
console.info(`[openhands] session.run roleId=${roleId} brief=${briefId ?? "none"}`);
|
||||
|
||||
try {
|
||||
const { conversationId } = await startConversation();
|
||||
console.info(`[openhands] conversation started id=${conversationId}`);
|
||||
|
||||
// Send the actual task prompt as a user message
|
||||
await sendMessage(conversationId, prompt);
|
||||
|
||||
// Poll until complete
|
||||
const result = await pollUntilComplete(conversationId);
|
||||
console.info(
|
||||
`[openhands] session complete status=${result.status} tools=${result.toolCalls} tokens=${result.tokensTotal ?? "unknown"}`,
|
||||
);
|
||||
|
||||
return {
|
||||
summary: result.summary,
|
||||
toolCalls: result.toolCalls,
|
||||
tokensTotal: result.tokensTotal,
|
||||
};
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
console.error(`[openhands] session failed:`, msg);
|
||||
return {
|
||||
summary: `OpenHands session failed: ${msg}`,
|
||||
toolCalls: 0,
|
||||
};
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
490
docs/AGENT-AUTONOMOUS-CONVOYS.md
Normal file
490
docs/AGENT-AUTONOMOUS-CONVOYS.md
Normal file
|
|
@ -0,0 +1,490 @@
|
|||
# Autonomous Convoys with OpenHands & OmniRoute
|
||||
|
||||
**Status:** Draft (planning phase)
|
||||
**Date:** 2026-09-07
|
||||
**Goal:** Enable OpenHands + omniroute to autonomously execute Echodo convoys by mapping roles to model profiles and integrating Agent Skills format.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Echodo's convoy system (defined in `.convoys/*.md` with role-based `model_policy`) currently requires Cursor to execute per-role briefs. This plan replaces Cursor with OpenHands as the runtime harness, using omniroute's model profiles (`auto`, `auto/coding`, `auto/reasoning`, `auto/coding-pro`, `auto/reasoning-pro`) to route each role to the appropriate model.
|
||||
|
||||
The main insight: **Your convoy structure already works. We're just swapping the agent harness and making skills discoverable.**
|
||||
|
||||
**Key changes:**
|
||||
1. Convert role-prompt rules to `.agents/skills/` in SKILL.md format (Agent Skills spec)
|
||||
2. Map omniroute profiles → roles (conductor/architect/implementer/reviewer)
|
||||
3. Wire OpenHands session bootstrap to inherit `agent_prompt` + skills + model policy
|
||||
4. Document the flow so agents executing convoys follow consistent patterns
|
||||
|
||||
**Outcome:** By end of v0.1, a human can:
|
||||
- Define a convoy in `.convoys/<slug>.md` with `model_policy` roles mapped to omniroute profiles
|
||||
- Trigger the convoy from Mission Control
|
||||
- OpenHands picks up each brief, inherits the task context, loads the right model via omniroute, and executes
|
||||
- After success, worker logs role completion + model metadata back to `agent_runs` and `convoy_events`
|
||||
|
||||
---
|
||||
|
||||
## Architecture: The Convoy-to-OpenHands Flow
|
||||
|
||||
```
|
||||
┌─── Human / Mission Control ──────────────────────────────────────┐
|
||||
│ "Start convoy: brief-1-implement-feature" │
|
||||
│ (.convoys/feature-x.md, brief-1-*.md exists, ready to go) │
|
||||
└────────────────────────┬────────────────────────────────────────┘
|
||||
│
|
||||
Orchestrator
|
||||
(apps/orchestrator)
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Redis Job Queue (echodo:orchestrator:jobs) │
|
||||
│ { roleId, harnessAlias, briefPath, workspaceId, ...} │
|
||||
└──────────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
Worker dequeues
|
||||
(apps/worker)
|
||||
│
|
||||
┌─────────────────┼──────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ Resolve │ │ Resolve │ │ Load │
|
||||
│ model via │ │ harness + │ │ session │
|
||||
│ omniroute │ │ worktree │ │ bootstrap: │
|
||||
│ profile │ │ │ │ • agent_prompt
|
||||
│ (coding, │ │ (code/doc/ │ │ • skills from
|
||||
│ reasoning) │ │ research │ │ .agents/ │
|
||||
│ │ │ harness) │ │ • role rules │
|
||||
└────────────┘ └──────────────┘ └──────────────┘
|
||||
│ │ │
|
||||
└────────────────┼──────────────────┘
|
||||
│
|
||||
OpenHands Session Start
|
||||
(apps/web calls MCP claim_task)
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Session context ready: │
|
||||
│ • Model resolved (e.g. qwen-coder or claude-3-5-sonnet) │
|
||||
│ • Task/brief context loaded (agent_prompt, file paths) │
|
||||
│ • Skills available (.agents/skills/*.md files) │
|
||||
│ • Tool allowlist set (code vs doc harness) │
|
||||
│ • MCP tools: read_file, write_file, grep, run_command │
|
||||
│ │
|
||||
│ Agent now thinks: "I am role-implementer on brief-1. │
|
||||
│ My success metric is: [from convoy frontmatter]. │
|
||||
│ I should follow: [agent_prompt from task]. │
|
||||
│ I can use these skills: [loaded from .agents/skills/]." │
|
||||
└────────────────────┬─────────────────────────────────────────┘
|
||||
│
|
||||
Agent tool loop
|
||||
(read_file, write_file,
|
||||
grep, run_command)
|
||||
│
|
||||
┌───────────────┼───────────────┐
|
||||
│ │ │
|
||||
Success Failure Stall
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||||
│ Log │ │ Log │ │ Orchestrator │
|
||||
│ complete │ │ failure │ │ detects │
|
||||
│ + model │ │ + │ │ stall, │
|
||||
│ metadata │ │ tokens │ │ resumes from │
|
||||
│ │ │ │ │ checkpoint │
|
||||
└────┬─────┘ └────┬─────┘ └──────┬───────┘
|
||||
│ │ │
|
||||
└─────────────┼───────────────┘
|
||||
│
|
||||
MCP complete_task + convoy_events.role_complete
|
||||
│
|
||||
Orchestrator ticks
|
||||
(determines next role)
|
||||
│
|
||||
┌─────────────┼──────────────┐
|
||||
│ │ │
|
||||
Next Brief Human Gate End Convoy
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌─────────┐ ┌──────────┐
|
||||
│ Back to │ │ Mission │ │ Released │
|
||||
│ Redis │ │ Control │ │ or │
|
||||
│ (loop) │ │ approve │ │ merged │
|
||||
└──────────┘ └─────────┘ └──────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Four Concrete Work Phases
|
||||
|
||||
### Phase 1: Agent Skills Format Conversion (M estimate)
|
||||
|
||||
**Goal:** Make repo-integrated skills discoverable by OpenHands via SKILL.md format in `.agents/skills/`.
|
||||
|
||||
**Why:** Your current `.cursor/rules/*.mdc` files are Cursor-specific. OpenHands reads `.agents/skills/*/SKILL.md` (per Agent Skills spec). Converting moves you from Cursor-only to Cursor + OpenHands + any harness that supports Agent Skills.
|
||||
|
||||
**Tasks:**
|
||||
|
||||
1. **Create `.agents/skills/backlog-sync/SKILL.md`**
|
||||
- **When to invoke:** When working with `plans/`, `markdown_backlog_items`, or the Echodo markdown import
|
||||
- **What it teaches:** How backlog frontmatter maps to DB, how `pnpm --filter @tasks/database watch:markdown-backlog` works, the contract between `.md` and `markdown_backlog_items`
|
||||
- **Reference:** `.cursor/rules/markdown-backlog.mdc` + `packages/database/src/markdown-backlog/*.ts` + `docs/Glossary.md`
|
||||
- **Output format:** `SKILL.md` with frontmatter (name, description) + sections (When to Activate, Core Rules, Common Patterns, Examples)
|
||||
|
||||
2. **Create `.agents/skills/convoy-lifecycle/SKILL.md`**
|
||||
- **When to invoke:** When working with convoys, briefs, transitions, gates, or the state machine
|
||||
- **What it teaches:** Status enum (draft→ready→in_progress→pr_review→merged), gate actors (human-gate-1/2/3), MCP tools (create_convoy, transition_convoy_status, log_convoy_event), checkpoint/resume pattern
|
||||
- **Reference:** `docs/convoy-lifecycle.md` + `packages/shared/src/convoy-lifecycle/*.ts` + `apps/mcp-server/src/tools/*convoy*`
|
||||
- **Output format:** Same as backlog-sync
|
||||
|
||||
3. **Create `.agents/skills/model-routing/SKILL.md`**
|
||||
- **When to invoke:** When routing models, configuring aliases, or understanding how the orchestrator selects models
|
||||
- **What it teaches:** The resolution order (brief override → convoy policy → workspace routes → pipeline defaults), omniroute profile mapping, how `resolveModel(workspace, roleId)` works, how model metadata flows to `agent_runs` and `convoy_events`
|
||||
- **Reference:** `docs/adr-model-routing.md` + `packages/shared/src/orchestrator/*.ts` + this doc's **Phase 2** section
|
||||
- **Output format:** Same as above
|
||||
|
||||
4. **Create `.agents/skills/code-review-two-axis/SKILL.md`** (adapted from mattpocock/skills)
|
||||
- **When to invoke:** When `role-reviewer` runs or a human asks for a PR review in Mission Control
|
||||
- **What it teaches:** Two-axis review (Standards vs Spec), running both as parallel sub-agents, how to aggregate findings, the code-smell baseline
|
||||
- **Reference:** https://github.com/mattpocock/skills/blob/main/skills/engineering/code-review/SKILL.md (with Echodo-specific tweaks for Standards against `AGENTS.md` + repo rules)
|
||||
- **Output format:** Same as above
|
||||
|
||||
5. **Create `.agents/skills/tdd-red-green/SKILL.md`** (adapted from mattpocock/skills)
|
||||
- **When to invoke:** When implementing features or fixing bugs in a test-first flow
|
||||
- **What it teaches:** Red-green-refactor loop, seams, anti-patterns, the tracer-bullet concept
|
||||
- **Reference:** https://github.com/mattpocock/skills/blob/main/skills/engineering/tdd/SKILL.md
|
||||
- **Output format:** Same as above
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Each SKILL.md has frontmatter (name, description) + at least 3 sections
|
||||
- [ ] `skil --validate .agents/skills/*/SKILL.md` passes (or equiv. local validation)
|
||||
- [ ] Skills are referenced in `AGENTS.md` under the "Skills available" section
|
||||
- [ ] Agent can invoke them via `/skill` command or model-invocation within OpenHands
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: OmniRoute Profile Mapping & Model Policy Doc (S estimate)
|
||||
|
||||
**Goal:** Document how omniroute profiles map to roles and how the orchestrator uses `model_policy` to route.
|
||||
|
||||
**Why:** Your `model_policy.roles` in convoy frontmatter lists role names + model aliases. Those aliases need to resolve to omniroute **profiles** (e.g., `auto/coding`, `auto/reasoning`) so the worker can ask omniroute "give me the model for `claude-planning`" and get back the current endpoint.
|
||||
|
||||
**Tasks:**
|
||||
|
||||
1. **Create `docs/omniroute-role-mapping.md`**
|
||||
- **Section 1: Profile inventory**
|
||||
```
|
||||
- auto (default, fastest)
|
||||
- auto/coding (code-specialized)
|
||||
- auto/coding-pro (complex, high-quality)
|
||||
- auto/reasoning (reasoning/planning)
|
||||
- auto/reasoning-pro (complex reasoning, high-cost)
|
||||
```
|
||||
- **Section 2: Role → Profile mapping (recommended)**
|
||||
```
|
||||
role-conductor → auto/reasoning (kickoff, classify, skip flags)
|
||||
role-architect → auto/reasoning-pro (design, brief decomposition)
|
||||
role-implementer → auto/coding (default) or auto/coding-pro (escalation)
|
||||
role-reviewer → auto/coding-pro (two-axis review)
|
||||
role-doc-writer → auto/coding (docs are simpler than code)
|
||||
gate-copilot → auto/reasoning (summary for human gate)
|
||||
```
|
||||
- **Section 3: How to configure in a convoy**
|
||||
```yaml
|
||||
model_policy:
|
||||
roles:
|
||||
role-conductor: claude-planning # resolves to auto/reasoning profile
|
||||
role-implementer: qwen-coder # resolves to auto/coding profile
|
||||
escalate_to: claude-planning # if implementer fails 2x, re-dispatch with this
|
||||
```
|
||||
- **Section 4: How the orchestrator uses this**
|
||||
- Reads `model_policy.roles[roleId]` from the convoy
|
||||
- Calls `resolveModel(workspace, roleId)` in `packages/shared/src/orchestrator/*.ts`
|
||||
- Resolution order: brief override → convoy policy → workspace routes → pipeline defaults
|
||||
- Result: `{ alias, provider_type, model_id, base_url, api_key_env_var }`
|
||||
- Worker receives model spec in the job payload
|
||||
- **Section 5: Escalation & retry logic**
|
||||
- If implementer brief fails, orchestrator can re-dispatch with `escalate_to: role-architect` (higher-tier model)
|
||||
- Log tier change on `agent_runs`
|
||||
|
||||
2. **Create `docs/omniroute-setup.md`** (operational guide)
|
||||
- **How to register omniroute in the workspace** (via Settings → AI → Providers or tRPC `model_providers.create`)
|
||||
- **How to set up the 5 profiles** (assuming omniroute cloud endpoint; or local Ollama + MLX)
|
||||
- **How to test** (`testProvider` MCP call, test a single model)
|
||||
- **How to debug** (logs in `agent_runs`, `worker_checkpoints` metadata)
|
||||
|
||||
3. **Update `docs/adr-model-routing.md`** (add OmniRoute section)
|
||||
- Keep the existing conceptual model
|
||||
- Add subsection: "OmniRoute Cloud Integration (v0.1)"
|
||||
- Explain how `resolveModel` calls omniroute's API to get the actual endpoint + auth
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] `docs/omniroute-role-mapping.md` covers all 6 roles (conductor, architect, implementer, reviewer, doc-writer, gate-copilot)
|
||||
- [ ] Each role has a recommended default profile + an override example
|
||||
- [ ] `docs/omniroute-setup.md` has step-by-step instructions (create alias, test, configure workspace route)
|
||||
- [ ] ADR updated with omniroute context
|
||||
- [ ] Convoy examples in `.convoys/*.md` use the recommended aliases (no hardcoded model names)
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: OpenHands Session Bootstrap & Worker Integration (L estimate)
|
||||
|
||||
**Goal:** Wire up OpenHands session to inherit task context, load skills, and respect model policy.
|
||||
|
||||
**Why:** When the worker dequeues a job and spawns an OpenHands session, the agent needs to know:
|
||||
- What task it's working on (brief path, success metric, acceptance criteria)
|
||||
- What role it is (role-implementer, role-reviewer, etc.)
|
||||
- Which model to use (resolved from omniroute)
|
||||
- Which skills are available (from `.agents/skills/`)
|
||||
- Which tools are allowed (code harness: `pnpm lint`, `run_command`; doc harness: read/write/grep only)
|
||||
|
||||
**Tasks:**
|
||||
|
||||
1. **Extend `apps/worker/src/session/minimal-session.ts`** (or create `openhands-session.ts`)
|
||||
- **Input:** `{ jobSpec, resolvedModel, harnessAlias, briefPath }`
|
||||
- **On session init:**
|
||||
- Load `brief-*.md` from `.convoys/<slug>/` (task context)
|
||||
- Extract `agent_prompt` (from task → epic → plan inheritance, per AGENTS.md)
|
||||
- Inherit `title`, `success_metric`, acceptance criteria from frontmatter
|
||||
- Construct a **session bootstrap** (initial system + first user message)
|
||||
- **Session bootstrap structure:**
|
||||
```
|
||||
System message (shared context):
|
||||
- You are an AI assistant helping implement a software task
|
||||
- Your task: [title from brief frontmatter]
|
||||
- Success means: [success_metric]
|
||||
- You are playing the role of: [roleId]
|
||||
- Acceptance criteria: [from brief or inherited]
|
||||
- Constraints:
|
||||
- Use the [tool allowlist] only
|
||||
- If you make a file change, it will be pushed to a branch
|
||||
- After you succeed, claim_task + complete_task will be called
|
||||
- Principles: [karpathy-guidelines from AGENTS.md]
|
||||
|
||||
First user message:
|
||||
- [inherited agent_prompt from task frontmatter]
|
||||
- [brief body content — the actual work description]
|
||||
- "Here are the skills you have available: [list from .agents/skills/]"
|
||||
- "Your model is: [resolved model name from omniroute]"
|
||||
```
|
||||
- **On session end:**
|
||||
- Collect metrics: `{ model, tokens, duration, outcome }`
|
||||
- Return to worker for logging
|
||||
|
||||
2. **Wire OpenHands integration into Worker job handler** (`apps/worker/src/handle-job.ts`)
|
||||
- **Current flow:** `MinimalSession` → tool loop → checkpoint
|
||||
- **New flow:**
|
||||
```
|
||||
1. Dequeue job from Redis
|
||||
2. resolveHarnessForJob(job.roleId) → harness config
|
||||
3. resolveModel(workspace, job.roleId) → model spec
|
||||
4. Create OpenHands session with bootstrap (brief context + skills)
|
||||
5. Run tool loop (agent interacts with codebase)
|
||||
6. On success: push branch, call forge PR endpoint
|
||||
7. Log complete_task + convoy_events.role_complete with model metadata
|
||||
8. Return to step 1
|
||||
```
|
||||
- **Config:** Add env var `WORKER_SESSION_TYPE=openai-client | minimal | openhands` (default: minimal for backward compat, set to `openhands` to use new flow)
|
||||
|
||||
3. **Create `.agents/` integration test** (`apps/worker/src/openhands-integration.test.ts`)
|
||||
- **Test:** Spawn a dry-run OpenHands session with a toy brief, verify:
|
||||
- [ ] Brief context loads correctly
|
||||
- [ ] Skills are discoverable
|
||||
- [ ] agent_prompt is inherited
|
||||
- [ ] Tool allowlist is respected
|
||||
- [ ] Session logs metrics on exit
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] `WORKER_SESSION_TYPE=openhands` env var enables the flow
|
||||
- [ ] Integration test passes (dry-run with mock brief)
|
||||
- [ ] Live E2E test (real brief, real repo) succeeds with model logging
|
||||
- [ ] `agent_runs` row includes `{ model_alias, model_tier, tokens_in, tokens_out }`
|
||||
- [ ] `convoy_events` has `role_complete` entry with `metadata.harnessAlias`
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: End-to-End E2E Test & Dogfood (L estimate)
|
||||
|
||||
**Goal:** Prove one convoy executes autonomously from start to finish using OpenHands + omniroute.
|
||||
|
||||
**Why:** This is the acceptance test for the whole milestone. A human should be able to:
|
||||
1. Define a convoy with roles mapped to omniroute profiles
|
||||
2. Click "Start" in Mission Control
|
||||
3. Watch OpenHands agents pick up and complete briefs without Cursor
|
||||
4. See model metadata logged correctly
|
||||
|
||||
**Tasks:**
|
||||
|
||||
1. **Create a test convoy** (`.convoys/test-e2e-autonomous.md`)
|
||||
- Classification: `server-only` (simple, no IA/UX/design)
|
||||
- 2–3 briefs (implementer + reviewer)
|
||||
- Roles: `role-conductor` (kickoff only), `role-implementer` (fix a small bug or add a minor feature), `role-reviewer` (review the PR)
|
||||
- Model policy: use omniroute profiles (e.g., `role-implementer: auto/coding`, `role-reviewer: auto/coding-pro`)
|
||||
- Success metric: "Bug is fixed, PR is drafted, review is complete"
|
||||
|
||||
2. **Run the full loop** (Manual walkthrough)
|
||||
- [ ] Human creates convoy via MCP `create_convoy` or Mission Control UI
|
||||
- [ ] Orchestrator picks up `role-conductor` (dry-run or light task)
|
||||
- [ ] Orchestrator enqueues `role-implementer` to Redis
|
||||
- [ ] Worker dequeues, spawns OpenHands session
|
||||
- [ ] OpenHands loads brief, runs tool loop, fixes the bug, commits
|
||||
- [ ] Worker logs `complete_task` + `role_complete`
|
||||
- [ ] Orchestrator transitions convoy to `pr_review` (or next gate)
|
||||
- [ ] Orchestrator enqueues `role-reviewer` brief
|
||||
- [ ] Worker dequeues, spawns OpenHands session, runs two-axis review
|
||||
- [ ] Worker logs `complete_task` + `role_complete` with review metadata
|
||||
- [ ] Convoy reaches `merged` or `published` gate
|
||||
- [ ] Check `agent_runs`: model aliases logged for both implementer + reviewer
|
||||
- [ ] Check `convoy_events`: role_complete entries present with metadata
|
||||
|
||||
3. **Dogfood the pipeline** (Optional: use this convoy to close a real task)
|
||||
- Pick a backlog task that fits the convoy (small feature or bug)
|
||||
- Map task to a convoy brief
|
||||
- Run the E2E flow on real code
|
||||
- Validate that the fix/feature is correct and the PR is mergeable
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] Convoy completes all briefs without Cursor
|
||||
- [ ] `agent_runs` has entries for implementer + reviewer with model aliases
|
||||
- [ ] `convoy_events` has `role_complete` entries
|
||||
- [ ] PR is draft or merged (depending on gates)
|
||||
- [ ] No manual intervention after "Start Convoy"
|
||||
- [ ] Logs are clean (no auth errors, model resolution errors)
|
||||
|
||||
---
|
||||
|
||||
## Dependencies & Ordering
|
||||
|
||||
```
|
||||
Phase 1 (Skills) ──┐
|
||||
Phase 2 (Routing) ─┼──→ Phase 3 (Worker) ──→ Phase 4 (E2E)
|
||||
Phase 3 unblocked │ (can start once 1 + 2 done)
|
||||
as soon as 1 + 2 │
|
||||
complete
|
||||
```
|
||||
|
||||
**Critical path:**
|
||||
- Phase 1 + 2 are sequential (skills first, then routing docs)
|
||||
- Phase 3 can start once Phase 1 + 2 are done
|
||||
- Phase 4 requires Phase 3 complete
|
||||
|
||||
---
|
||||
|
||||
## Agent Instructions for Each Phase
|
||||
|
||||
### For Agents Working on Phase 1 (Skills)
|
||||
|
||||
1. **Before writing:** Read the existing `.cursor/rules/*.mdc` files in this repo + the referenced SKILL.md files from mattpocock/skills, anthropics/skills
|
||||
2. **Structure:** Follow the Agent Skills spec (frontmatter + markdown sections)
|
||||
3. **Validation:** Each skill should be invokable by model (e.g., `/skill backlog-sync` or implied when needed) and have 3–4 substantive sections
|
||||
4. **Test:** Run locally with OpenHands or Claude Code to verify skills load
|
||||
5. **Acceptance:** PR reviewer checks that skills have frontmatter + clear When-to-Activate section
|
||||
|
||||
### For Agents Working on Phase 2 (Routing)
|
||||
|
||||
1. **Before writing:** Read `docs/adr-model-routing.md` fully + understand how `model_policy` works in `.convoys/*.md`
|
||||
2. **Assumption:** OmniRoute cloud is the target (not local-only), and omniroute has 5 profiles (auto, auto/coding, auto/coding-pro, auto/reasoning, auto/reasoning-pro)
|
||||
3. **Output:** Two markdown files (mapping + setup), both human-readable and copy-pasteable examples
|
||||
4. **Validation:** Walk through the flow: convoy sets `model_policy.roles`, orchestrator calls `resolveModel`, worker receives model spec
|
||||
5. **Test:** No automated test needed; human walkthrough in Phase 4
|
||||
|
||||
### For Agents Working on Phase 3 (Worker Integration)
|
||||
|
||||
1. **Before writing:** Read `apps/worker/src/session/minimal-session.ts` + `apps/worker/src/handle-job.ts` + understand the worker dequeue loop
|
||||
2. **Pattern:** Don't replace MinimalSession; extend it or wrap it in an adapter (prefer wrapping for backward compat)
|
||||
3. **Assumption:** OpenHands agent-server is running and reachable (via env var `AGENT_SERVER_URL`)
|
||||
4. **Validation:**
|
||||
- Type-check passes (`pnpm type-check`)
|
||||
- New env var `WORKER_SESSION_TYPE` is optional (defaults to `minimal`)
|
||||
- Integration test dry-runs without errors
|
||||
5. **Test:** Phase 4 (E2E) will exercise this
|
||||
|
||||
### For Agents Working on Phase 4 (E2E)
|
||||
|
||||
1. **Before starting:** Phase 1 + 2 + 3 must be done
|
||||
2. **Pre-flight:**
|
||||
- [ ] `.agents/skills/` SKILL.md files exist + load correctly
|
||||
- [ ] `docs/omniroute-role-mapping.md` + `docs/omniroute-setup.md` exist
|
||||
- [ ] `WORKER_SESSION_TYPE=openhands` env var works
|
||||
- [ ] Orchestrator + worker are running (or can dry-run)
|
||||
3. **Test convoy:** Create `.convoys/test-e2e-autonomous.md` with 2 briefs (implementer + reviewer)
|
||||
4. **Manual walkthrough:** Follow the flow in "Phase 4: Tasks" section, step by step
|
||||
5. **Validation:** Check `agent_runs`, `convoy_events` tables for logs; verify PR created + reviewed
|
||||
6. **Report:** Summarize the run (timeline, model aliases used, any errors, fixes applied)
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria (Overall)
|
||||
|
||||
By end of Phase 4:
|
||||
|
||||
- [ ] One full convoy (2+ briefs) executes autonomously via OpenHands without Cursor
|
||||
- [ ] Each role uses the model from omniroute (via model_policy)
|
||||
- [ ] `agent_runs` + `convoy_events` log model metadata for observability
|
||||
- [ ] No secrets or API keys leaked in logs
|
||||
- [ ] Docs cover setup + usage (omniroute-role-mapping.md, omniroute-setup.md)
|
||||
- [ ] Skills are discoverable + usable by agents (.agents/skills/*/SKILL.md)
|
||||
- [ ] Worker integration test passes
|
||||
- [ ] Code is linted + type-checked
|
||||
|
||||
---
|
||||
|
||||
## Rollout & Scaling
|
||||
|
||||
Once Phase 4 succeeds:
|
||||
|
||||
1. **Publish to v0.1** — Mark the orchestrator + worker as production-ready for autonomous convoys
|
||||
2. **Scale briefs** — Extend to 5–6 briefs per convoy (architect → implementer → reviewer → ops-copilot → etc.)
|
||||
3. **Scale roles** — Add `role-docs-writer`, `role-security-auditor`, `role-performance-reviewer` as needed
|
||||
4. **Scale models** — Extend omniroute integration to support Claude/OpenAI/Gemini APIs side-by-side
|
||||
5. **Migrate Cursor convoys** — For convoys that have historically run in Cursor, run them via OpenHands instead (backward compat: human-initiated, not automated)
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Model tier mismatches (expensive model for simple task) | OmniRoute profiles + explicit role mapping in docs |
|
||||
| Agent hallucinations in brief context | Agent_prompt inheritance from task → epic → plan keeps context tight; karpathy-guidelines skill |
|
||||
| Network issues between worker + OpenHands | Checkpoint/resume logic already in place; retry logic in orchestrator |
|
||||
| Secrets leaked in logs | Review `agent_runs.metadata` + `worker_checkpoints.tool_history` before logging; never log full API responses |
|
||||
| Convoy state desync (DB vs `.convoys/*.md`) | `reconcile_from_files` MCP tool keeps them in sync; convoy_events is the source of truth |
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Agent Skills Spec:** https://agentskills.io/specification
|
||||
- **Mattpocock Skills (code-review, tdd):** https://github.com/mattpocock/skills
|
||||
- **Superpowers (bootstrap pattern):** https://github.com/obra/superpowers
|
||||
- **Karpathy Guidelines (4 behavioral rules):** https://github.com/multica-ai/andrej-karpathy-skills
|
||||
- **Echodo Convoys:** `.convoys/README.md`
|
||||
- **Model Routing ADR:** `docs/adr-model-routing.md`
|
||||
- **Worker Docs:** `docs/worker.md`
|
||||
- **Orchestrator Docs:** `docs/orchestrator.md`
|
||||
|
||||
---
|
||||
|
||||
## Questions for Review
|
||||
|
||||
Before agents start Phase 1, answer these:
|
||||
|
||||
1. **OmniRoute target:** Are we assuming omniroute cloud endpoint, or should we support local Ollama + MLX as a fallback?
|
||||
2. **Model fallback:** If omniroute is down, should the worker use a hardcoded fallback model, or fail closed?
|
||||
3. **Role coverage:** Are 6 roles (conductor, architect, implementer, reviewer, doc-writer, gate-copilot) enough, or do we need more?
|
||||
4. **Cost tracking:** Should Phase 3 also wire up `cost_budget` checks from `model_providers`, or leave that for a follow-on?
|
||||
5. **Backward compat:** Should Phase 3 preserve the MinimalSession adapter, or fully replace it with OpenHands?
|
||||
6. **v0.1 scope:** Phase 4 tests one convoy. Should we aim for 2+ convoys running in parallel by end of Phase 4, or save that for post-v0.1?
|
||||
|
||||
---
|
||||
|
||||
**Document Version:** 1.0
|
||||
**Last Updated:** 2026-09-07
|
||||
**Next Review:** After Phase 1 completion
|
||||
313
docs/omniroute-role-mapping.md
Normal file
313
docs/omniroute-role-mapping.md
Normal file
|
|
@ -0,0 +1,313 @@
|
|||
# OmniRoute Role Mapping
|
||||
|
||||
**Status:** Draft
|
||||
**Date:** 2026-09-07
|
||||
**Purpose:** Map Echodo pipeline roles to OmniRoute profiles so the orchestrator/worker can route each role to the right model automatically.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
OmniRoute is an OpenAI-compatible gateway that fronts **929+ models** with a small set of **dynamic profiles**. Instead of configuring individual model IDs per role, we configure **profiles** that OmniRoute resolves to a live model endpoint:
|
||||
|
||||
| Profile | Purpose | Typical Resolves To |
|
||||
|---------|---------|---------------------|
|
||||
| `auto` | Fastest general-purpose | Small/fast model |
|
||||
| `auto/coding` | Code-specialized | Coding-tuned model |
|
||||
| `auto/coding-pro` | Complex code tasks, reviews | High-quality coding model |
|
||||
| `auto/reasoning` | Planning, reasoning, architecture | Reasoning-tuned model |
|
||||
| `auto/reasoning-pro` | Complex reasoning, escalation | Top reasoning model |
|
||||
|
||||
OmniRoute is integrated into Echodo as an **`openai_compatible` provider** — the same way Ollama and MLX are, but pointing at the OmniRoute base URL and using profile names as `modelId`.
|
||||
|
||||
---
|
||||
|
||||
## Profile → Role Matrix
|
||||
|
||||
### Recommended Defaults
|
||||
|
||||
| Role ID | OmniRoute Profile | Why |
|
||||
|---------|-------------------|-----|
|
||||
| `role-conductor` | `auto/reasoning` | Convoy kickoff: classify, skip flags, success metric |
|
||||
| `role-architect` | `auto/reasoning` | Design, brief decomposition, architecture |
|
||||
| `role-architect` (escalation) | `auto/reasoning-pro` | Complex/hard design problems |
|
||||
| `role-ia-architect` | `auto/reasoning` | Information architecture planning |
|
||||
| `role-implementer` | `auto/coding` | Default code implementation |
|
||||
| `role-implementer` (escalation) | `auto/coding-pro` | After 2 failed attempts |
|
||||
| `role-reviewer` | `auto/coding-pro` | Two-axis code review needs quality |
|
||||
| `role-ux-reviewer` | `auto/coding` | UI review |
|
||||
| `role-doc-writer` | `auto/coding` | Docs are simpler than code |
|
||||
| `role-design-system-auditor` | `auto/coding` | Design system review |
|
||||
| `role-a11y-auditor` | `auto/coding` | Accessibility review |
|
||||
| `gate-copilot` | `auto/reasoning` | Summarize for human gate |
|
||||
| `chat-default` | `auto` | Web chat default |
|
||||
|
||||
### OmniRoute-Only Workspace (Simple Setup)
|
||||
|
||||
If a workspace does **not** use local models (no Ollama/MLX), just create **5 OmniRoute providers** (one per profile) and route every role to the relevant profile:
|
||||
|
||||
```yaml
|
||||
# Workspace model providers (via Settings → AI → Providers)
|
||||
- alias: omni-auto
|
||||
provider_type: openai_compatible
|
||||
model_id: auto
|
||||
base_url: ${OMNIROUTE_BASE_URL} # e.g. https://omniroute.example.com/v1
|
||||
api_key_env_var: OMNIROUTE_API_KEY
|
||||
metadata: { tier: auto, provider: omniroute }
|
||||
|
||||
- alias: omni-coding
|
||||
provider_type: openai_compatible
|
||||
model_id: auto/coding
|
||||
base_url: ${OMNIROUTE_BASE_URL}
|
||||
api_key_env_var: OMNIROUTE_API_KEY
|
||||
metadata: { tier: auto, provider: omniroute }
|
||||
|
||||
- alias: omni-coding-pro
|
||||
provider_type: openai_compatible
|
||||
model_id: auto/coding-pro
|
||||
base_url: ${OMNIROUTE_BASE_URL}
|
||||
api_key_env_var: OMNIROUTE_API_KEY
|
||||
metadata: { tier: auto, provider: omniroute }
|
||||
|
||||
- alias: omni-reasoning
|
||||
provider_type: openai_compatible
|
||||
model_id: auto/reasoning
|
||||
base_url: ${OMNIROUTE_BASE_URL}
|
||||
api_key_env_var: OMNIROUTE_API_KEY
|
||||
metadata: { tier: auto, provider: omniroute }
|
||||
|
||||
- alias: omni-reasoning-pro
|
||||
provider_type: openai_compatible
|
||||
model_id: auto/reasoning-pro
|
||||
base_url: ${OMNIROUTE_BASE_URL}
|
||||
api_key_env_var: OMNIROUTE_API_KEY
|
||||
metadata: { tier: auto, provider: omniroute }
|
||||
```
|
||||
|
||||
Then map roles:
|
||||
|
||||
| Role ID | Alias |
|
||||
|---------|-------|
|
||||
| `role-conductor` | `omni-reasoning` |
|
||||
| `role-architect` | `omni-reasoning` |
|
||||
| `role-ia-architect` | `omni-reasoning` |
|
||||
| `role-implementer` | `omni-coding` |
|
||||
| `role-reviewer` | `omni-coding-pro` |
|
||||
| `role-ux-reviewer` | `omni-coding` |
|
||||
| `role-doc-writer` | `omni-coding` |
|
||||
| `role-design-system-auditor` | `omni-coding` |
|
||||
| `role-a11y-auditor` | `omni-coding` |
|
||||
| `gate-copilot` | `omni-reasoning` |
|
||||
| `chat-default` | `omni-auto` |
|
||||
|
||||
### Hybrid Setup (Local + OmniRoute)
|
||||
|
||||
For a homelab that wants free local inference AND cloud fallback:
|
||||
|
||||
| Role ID | Default | Escalation |
|
||||
|---------|---------|------------|
|
||||
| `role-conductor` | `omni-reasoning` (cloud) | — |
|
||||
| `role-architect` | `omni-reasoning-pro` (cloud) | — |
|
||||
| `role-implementer` | `qwen-coder-mlx` (local) | `omni-coding-pro` (cloud) |
|
||||
| `role-reviewer` | `qwen-coder-mlx` (local) | `omni-coding-pro` (cloud) |
|
||||
| `role-ux-reviewer` | `qwen-coder-mlx` (local) | `omni-coding` |
|
||||
| `role-doc-writer` | `qwen-coder-mlx` (local) | `omni-coding` |
|
||||
| `gate-copilot` | `omni-reasoning` (cloud) | — |
|
||||
| `chat-default` | `omni-auto` (cloud) | — |
|
||||
|
||||
Hardcoded in convoy `model_policy`:
|
||||
|
||||
```yaml
|
||||
model_policy:
|
||||
roles:
|
||||
role-conductor: omni-reasoning
|
||||
role-architect: omni-reasoning-pro
|
||||
role-implementer: qwen-coder-mlx
|
||||
role-reviewer: omni-coding-pro
|
||||
role-doc-writer: qwen-coder-mlx
|
||||
escalate_to: omni-coding-pro
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How Resolution Works
|
||||
|
||||
Echodo's `resolveModel(workspaceId, roleId, ctx?)` (pure function in `packages/shared/src/model-routing/resolve-model.ts`) applies this order:
|
||||
|
||||
```
|
||||
1. Brief override → recommended_model in .convoys/<slug>/brief-*.md
|
||||
2. Convoy override → model_policy.roles[roleId] in .convoys/<slug>.md
|
||||
3. Workspace route → role_model_routes row in DB
|
||||
4. Pipeline template → PIPELINE_DEFAULT_ROLE_ROUTES (shipped defaults)
|
||||
```
|
||||
|
||||
The resolved `ResolvedModel` object is passed to the worker. The worker creates a Vercel AI SDK `LanguageModel` via `createLanguageModelFromResolved` (`packages/ai/src/providers/resolve-client.ts`).
|
||||
|
||||
**Because OmniRoute is `openai_compatible`:** it flows through the existing `case "openai_compatible"` branch, so **no code change is required** in `resolve-client.ts` — it uses `createOpenAI({ apiKey, baseURL })` with `modelId` = profile name.
|
||||
|
||||
---
|
||||
|
||||
## Resolved Model Example
|
||||
|
||||
For `role-implementer` → `omni-coding`:
|
||||
|
||||
```typescript
|
||||
{
|
||||
alias: "omni-coding",
|
||||
providerType: "openai_compatible",
|
||||
modelId: "auto/coding",
|
||||
baseUrl: "https://omniroute.example.com/v1",
|
||||
apiKey: "sk-...", // read from OMNIROUTE_API_KEY at runtime
|
||||
tier: "auto"
|
||||
}
|
||||
```
|
||||
|
||||
The worker's `createLanguageModelFromResolved` builds:
|
||||
|
||||
```typescript
|
||||
const openai = createOpenAI({
|
||||
apiKey: "sk-...",
|
||||
baseURL: "https://omniroute.example.com/v1",
|
||||
});
|
||||
const model = openai("auto/coding");
|
||||
```
|
||||
|
||||
OmniRoute then routes `auto/coding` to an appropriate live model per its own logic.
|
||||
|
||||
---
|
||||
|
||||
## Convoy Frontmatter
|
||||
|
||||
### OmniRoute-Only Convoy
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: feature-auth
|
||||
classification: server-only
|
||||
success_metric: POST /auth/register works and returns 201
|
||||
skip:
|
||||
- ia
|
||||
- ux
|
||||
- design
|
||||
- a11y
|
||||
- visual
|
||||
- smoke
|
||||
status: draft
|
||||
created: 2026-09-07
|
||||
model_policy:
|
||||
roles:
|
||||
role-conductor: omni-reasoning
|
||||
role-architect: omni-reasoning
|
||||
role-implementer: omni-coding
|
||||
role-reviewer: omni-coding-pro
|
||||
escalate_to: omni-coding-pro
|
||||
---
|
||||
```
|
||||
|
||||
### Hybrid Convoy
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: feature-local-first
|
||||
model_policy:
|
||||
roles:
|
||||
role-conductor: omni-reasoning
|
||||
role-architect: omni-reasoning
|
||||
role-implementer: qwen-coder-mlx
|
||||
role-reviewer: qwen-coder-mlx
|
||||
escalate_to: omni-coding-pro
|
||||
---
|
||||
```
|
||||
|
||||
### Brief Override
|
||||
|
||||
```yaml
|
||||
---
|
||||
brief: complex-db-migration
|
||||
recommended_model: omni-reasoning-pro # overrides convoy + workspace
|
||||
model_tier: auto
|
||||
---
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Where | Purpose |
|
||||
|----------|-------|---------|
|
||||
| `OMNIROUTE_BASE_URL` | Worker + web process env | OmniRoute endpoint (must end in `/v1`) |
|
||||
| `OMNIROUTE_API_KEY` | Worker + web process env | Autogenerated by OpenHands/OmniRoute |
|
||||
|
||||
These names are **referenced** in `model_providers.apiKeyEnvVar` — the actual values live in deploy env (Coolify/Dokploy/Mac), never in the DB or committed files.
|
||||
|
||||
---
|
||||
|
||||
## Testing the Mapping
|
||||
|
||||
### Via tRPC
|
||||
|
||||
```typescript
|
||||
// Debug endpoint: resolve a role to a model
|
||||
resolveForRole({ roleId: "role-implementer" })
|
||||
// → { ok: true, model: { alias: "omni-coding", modelId: "auto/coding", ... } }
|
||||
```
|
||||
|
||||
### Via MCP testProvider
|
||||
|
||||
```typescript
|
||||
testProvider({ alias: "omni-coding" })
|
||||
// → { success: true, latency_ms: 320, model: "auto/coding" }
|
||||
```
|
||||
|
||||
### Via direct OpenAI-compatible call
|
||||
|
||||
```bash
|
||||
curl ${OMNIROUTE_BASE_URL}/models \
|
||||
-H "Authorization: Bearer ${OMNIROUTE_API_KEY}"
|
||||
# → List of available models (929+ for OmniRoute)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Default Pipeline Template After OmniRoute
|
||||
|
||||
Update `PIPELINE_DEFAULT_ROLE_ROUTES` in `packages/shared/src/model-routing/constants.ts` if OmniRoute is the default for new workspaces:
|
||||
|
||||
```typescript
|
||||
export const PIPELINE_DEFAULT_ROLE_ROUTES: Readonly<Record<string, string>> = {
|
||||
"role-conductor": "omni-reasoning",
|
||||
"role-architect": "omni-reasoning",
|
||||
"role-ia-architect": "omni-reasoning",
|
||||
"role-ux-reviewer": "omni-coding",
|
||||
"role-implementer": "omni-coding",
|
||||
"role-reviewer": "omni-coding-pro",
|
||||
"role-design-system-auditor": "omni-coding",
|
||||
"role-a11y-auditor": "omni-coding",
|
||||
"role-doc-writer": "omni-coding",
|
||||
"gate-copilot": "omni-reasoning",
|
||||
"chat-default": "omni-auto",
|
||||
};
|
||||
```
|
||||
|
||||
*(Note: changing the template only affects new workspaces — existing workspace routes in the DB take precedence.)*
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Should the template switch to OmniRoute aliases by default, or stay Claude/Qwen and let operators pick OmniRoute?** Recommend: keep Claude/Qwen seed, document OmniRoute as an option until it's proven.
|
||||
2. **Does OmniRoute support streaming + function calling?** Needed for the worker tool loop. Verify via a live test before routing implementer roles to OmniRoute.
|
||||
3. **Rate limits / cost:** OmniRoute `auto/*` profiles rotate models — need token/cost tracking in `agent_runs` to monitor spend.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- **Resolver:** `packages/shared/src/model-routing/resolve-model.ts`
|
||||
- **Schemas:** `packages/shared/src/model-routing/schemas.ts`
|
||||
- **Constants:** `packages/shared/src/model-routing/constants.ts`
|
||||
- **Client builder:** `packages/ai/src/providers/resolve-client.ts`
|
||||
- **Provider schema:** `packages/database/src/schema/model_providers.ts`
|
||||
- **Route schema:** `packages/database/src/schema/role_model_routes.ts`
|
||||
- **ADR:** `docs/adr-model-routing.md`
|
||||
- **Skill:** `.agents/skills/model-routing/SKILL.md`
|
||||
384
docs/omniroute-setup.md
Normal file
384
docs/omniroute-setup.md
Normal file
|
|
@ -0,0 +1,384 @@
|
|||
# OmniRoute Setup Guide
|
||||
|
||||
**Status:** Draft
|
||||
**Date:** 2026-09-07
|
||||
**Purpose:** Step-by-step guide for operators to register OmniRoute as a model provider in Echodo, configure role routes, and test that everything works.
|
||||
|
||||
---
|
||||
|
||||
## What is OmniRoute?
|
||||
|
||||
OmniRoute is an OpenAI-compatible model routing gateway. It provides **dynamic model profiles** (`auto`, `auto/coding`, `auto/coding-pro`, `auto/reasoning`, `auto/reasoning-pro`) that automatically resolve to the best available model based on your OmniRoute plan and current load.
|
||||
|
||||
**Benefits for Echodo:**
|
||||
- Single API key, 929+ models available
|
||||
- Auto-routing: no need to pick individual models per role
|
||||
- Fallback/resilience: if one model is down, OmniRoute switches automatically
|
||||
- Works through Echodo's existing `openai_compatible` provider type — zero code changes required
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Echodo is running locally (`pnpm dev`) or deployed (Coolify/Dokploy)
|
||||
2. Database is initialized (`pnpm db:push` or `pnpm db:migrate`)
|
||||
3. You have an OmniRoute account/API key (or plan to use the local instance)
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Get OmniRoute Credentials
|
||||
|
||||
### Option A: OmniRoute Cloud (recommended)
|
||||
|
||||
1. Sign up at [omniroute.example.com](https://omniroute.example.com)
|
||||
2. Create an API key in Settings → API Keys
|
||||
3. Note your:
|
||||
- **Base URL:** `https://omniroute.example.com/v1` (check your dashboard)
|
||||
- **API Key:** `sk-...` (save securely — not committed to git)
|
||||
|
||||
### Option B: Self-hosted OmniRoute (local)
|
||||
|
||||
If running OmniRoute locally (e.g., via Docker):
|
||||
|
||||
1. Clone/run the OmniRoute server
|
||||
2. Note your:
|
||||
- **Base URL:** `http://localhost:11434/v1` (or whatever port)
|
||||
- **API Key:** `any-non-empty-string` (self-hosted often ignores auth)
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Set Environment Variables
|
||||
|
||||
Add to your `.env` (or Coolify environment):
|
||||
|
||||
```bash
|
||||
# OmniRoute credentials
|
||||
OMNIROUTE_BASE_URL=https://omniroute.example.com/v1
|
||||
OMNIROUTE_API_KEY=sk-your-key-here
|
||||
|
||||
# (Already set by default) Database
|
||||
DATABASE_URL=postgres://echodo:echodo@localhost:5432/echodo
|
||||
|
||||
# (Already set by default) Redis (for orchestrator + worker)
|
||||
REDIS_URL=redis://localhost:6379
|
||||
```
|
||||
|
||||
**Important:** These env vars must be available to:
|
||||
- The **web app** (`apps/web`) — for Settings UI
|
||||
- The **worker** (`apps/worker`) — for inference
|
||||
- The **orchestrator** (`apps/orchestrator`) — for dispatch
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Register OmniRoute Providers in Echodo
|
||||
|
||||
### Via Settings UI (recommended)
|
||||
|
||||
1. Open `http://localhost:3001` (web app)
|
||||
2. Go to **Settings → AI → Providers**
|
||||
3. Click **Add Provider** and create one for each profile:
|
||||
|
||||
**Provider 1: omni-auto**
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Display Name | `OmniRoute Auto` |
|
||||
| Alias | `omni-auto` |
|
||||
| Provider Type | `openai_compatible` |
|
||||
| Model ID | `auto` |
|
||||
| Base URL | `${OMNIROUTE_BASE_URL}` (your actual URL) |
|
||||
| API Key Env Var | `OMNIROUTE_API_KEY` |
|
||||
| Metadata | `{"tier": "auto", "provider": "omniroute"}` |
|
||||
| Enabled | ✅ |
|
||||
|
||||
**Provider 2: omni-coding**
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Display Name | `OmniRoute Coding` |
|
||||
| Alias | `omni-coding` |
|
||||
| Provider Type | `openai_compatible` |
|
||||
| Model ID | `auto/coding` |
|
||||
| Base URL | (same as above) |
|
||||
| API Key Env Var | `OMNIROUTE_API_KEY` |
|
||||
| Metadata | `{"tier": "auto", "provider": "omniroute"}` |
|
||||
| Enabled | ✅ |
|
||||
|
||||
**Provider 3: omni-coding-pro**
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Display Name | `OmniRoute Coding Pro` |
|
||||
| Alias | `omni-coding-pro` |
|
||||
| Provider Type | `openai_compatible` |
|
||||
| Model ID | `auto/coding-pro` |
|
||||
| Base URL | (same) |
|
||||
| API Key Env Var | `OMNIROUTE_API_KEY` |
|
||||
| Metadata | `{"tier": "auto", "provider": "omniroute"}` |
|
||||
| Enabled | ✅ |
|
||||
|
||||
**Provider 4: omni-reasoning**
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Display Name | `OmniRoute Reasoning` |
|
||||
| Alias | `omni-reasoning` |
|
||||
| Provider Type | `openai_compatible` |
|
||||
| Model ID | `auto/reasoning` |
|
||||
| Base URL | (same) |
|
||||
| API Key Env Var | `OMNIROUTE_API_KEY` |
|
||||
| Metadata | `{"tier": "auto", "provider": "omniroute"}` |
|
||||
| Enabled | ✅ |
|
||||
|
||||
**Provider 5: omni-reasoning-pro**
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Display Name | `OmniRoute Reasoning Pro` |
|
||||
| Alias | `omni-reasoning-pro` |
|
||||
| Provider Type | `openai_compatible` |
|
||||
| Model ID | `auto/reasoning-pro` |
|
||||
| Base URL | (same) |
|
||||
| API Key Env Var | `OMNIROUTE_API_KEY` |
|
||||
| Metadata | `{"tier": "auto", "provider": "omniroute"}` |
|
||||
| Enabled | ✅ |
|
||||
|
||||
### Via tRPC (for automation)
|
||||
|
||||
```typescript
|
||||
// In a script or test
|
||||
await trpc.ai.models.createProvider.mutate({
|
||||
alias: "omni-coding",
|
||||
displayName: "OmniRoute Coding",
|
||||
providerType: "openai_compatible",
|
||||
modelId: "auto/coding",
|
||||
baseUrl: "https://omniroute.example.com/v1",
|
||||
apiKeyEnvVar: "OMNIROUTE_API_KEY",
|
||||
metadata: { tier: "auto", provider: "omniroute" }
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Map Roles to Profiles
|
||||
|
||||
### Via Settings UI
|
||||
|
||||
1. Go to **Settings → AI → Role Routing**
|
||||
2. For each role, click **Edit** and select the provider:
|
||||
|
||||
| Role ID | Select Provider |
|
||||
|---------|-----------------|
|
||||
| `role-conductor` | `omni-reasoning` |
|
||||
| `role-architect` | `omni-reasoning` |
|
||||
| `role-ia-architect` | `omni-reasoning` |
|
||||
| `role-implementer` | `omni-coding` |
|
||||
| `role-reviewer` | `omni-coding-pro` |
|
||||
| `role-ux-reviewer` | `omni-coding` |
|
||||
| `role-doc-writer` | `omni-coding` |
|
||||
| `role-design-system-auditor` | `omni-coding` |
|
||||
| `role-a11y-auditor` | `omni-coding` |
|
||||
| `gate-copilot` | `omni-reasoning` |
|
||||
| `chat-default` | `omni-auto` |
|
||||
|
||||
### Via tRPC
|
||||
|
||||
```typescript
|
||||
// Example: route implementer to omni-coding
|
||||
await trpc.ai.models.upsertRoleRoute.mutate({
|
||||
roleId: "role-implementer",
|
||||
providerAliasId: "omni-coding-provider-uuid" // get from listProviders
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Test Connectivity
|
||||
|
||||
### Via Settings UI
|
||||
|
||||
1. Go to **Settings → AI → Providers**
|
||||
2. Click the **Test** button next to `omni-coding`
|
||||
3. You should see: ✅ `Success — latency: 320ms`
|
||||
|
||||
### Via MCP (if worker is running)
|
||||
|
||||
```typescript
|
||||
// testProvider
|
||||
{
|
||||
"tool": "testProvider",
|
||||
"arguments": { "alias": "omni-coding" }
|
||||
}
|
||||
```
|
||||
|
||||
### Via direct curl
|
||||
|
||||
```bash
|
||||
curl ${OMNIROUTE_BASE_URL}/chat/completions \
|
||||
-H "Authorization: Bearer ${OMNIROUTE_API_KEY}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto/coding",
|
||||
"messages": [{"role": "user", "content": "Say hello in 3 words"}],
|
||||
"max_tokens": 20
|
||||
}'
|
||||
```
|
||||
|
||||
Expected: A response like `{"choices": [{"message": {"content": "Hello world!"}}]}`
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Test Model Resolution
|
||||
|
||||
### Verify a specific role resolves correctly
|
||||
|
||||
Use the debug endpoint (or MCP `resolveForRole`):
|
||||
|
||||
```typescript
|
||||
resolveForRole({ roleId: "role-implementer" })
|
||||
// Expected output:
|
||||
// {
|
||||
// ok: true,
|
||||
// model: {
|
||||
// alias: "omni-coding",
|
||||
// providerType: "openai_compatible",
|
||||
// modelId: "auto/coding",
|
||||
// baseUrl: "https://omniroute.example.com/v1",
|
||||
// apiKey: "<redacted>",
|
||||
// tier: "auto"
|
||||
// },
|
||||
// source: "workspace",
|
||||
// resolvedAlias: "omni-coding"
|
||||
// }
|
||||
```
|
||||
|
||||
### Verify resolution order works
|
||||
|
||||
Create a test convoy with a model override:
|
||||
|
||||
```yaml
|
||||
# .convoys/test-resolver.md
|
||||
---
|
||||
name: test-resolver
|
||||
model_policy:
|
||||
roles:
|
||||
role-implementer: omni-reasoning-pro # overrides workspace route
|
||||
---
|
||||
```
|
||||
|
||||
Then `resolveForRole` with `convoyContext`:
|
||||
|
||||
```typescript
|
||||
resolveForRole({
|
||||
roleId: "role-implementer",
|
||||
convoyContext: { model_policy: { "role-implementer": "omni-reasoning-pro" } }
|
||||
})
|
||||
// → source: "convoy", alias: "omni-reasoning-pro"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Verify Worker Picks Up the Model
|
||||
|
||||
1. **Start the worker:**
|
||||
```bash
|
||||
pnpm --filter @tasks/worker dev
|
||||
# or
|
||||
WORKER_DRY_RUN=0 ORCHESTRATOR_DRY_RUN=0 REDIS_URL=redis://localhost:6379 pnpm --filter @tasks/worker dev
|
||||
```
|
||||
|
||||
2. **Start the orchestrator:**
|
||||
```bash
|
||||
pnpm --filter @tasks/orchestrator dev
|
||||
```
|
||||
|
||||
3. **Enqueue a test job** (via MCP `create_convoy` → transition to `ready`)
|
||||
|
||||
4. **Watch worker logs:**
|
||||
- Should see: `Dequeued job for role-implementer, resolved model: omni-coding (auto/coding)`
|
||||
- Should see: inference call to OmniRoute endpoint
|
||||
|
||||
5. **Verify model metadata in `agent_runs`:**
|
||||
```sql
|
||||
SELECT model_alias, provider_type, model_id, model_tier
|
||||
FROM agent_runs
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 5;
|
||||
```
|
||||
Expected: `omni-coding | openai_compatible | auto/coding | auto`
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Provider alias 'omni-coding' is not registered in this workspace"
|
||||
|
||||
**Cause:** Provider not created yet, or `alias` doesn't match.
|
||||
|
||||
**Fix:**
|
||||
1. Check Settings → AI → Providers for the alias name
|
||||
2. Re-run `createProvider` with exact alias
|
||||
|
||||
### "Environment variable OMNIROUTE_API_KEY is not set"
|
||||
|
||||
**Cause:** Worker process doesn't have the env var loaded.
|
||||
|
||||
**Fix:**
|
||||
1. Verify `OMNIROUTE_API_KEY` is in `.env` at repo root
|
||||
2. Restart the worker process (env vars aren't hot-reloaded)
|
||||
3. Check: `echo $OMNIROUTE_API_KEY` in the worker terminal
|
||||
|
||||
### "Connection refused" or "ECONNREFUSED"
|
||||
|
||||
**Cause:** OmniRoute server not reachable at `base_url`.
|
||||
|
||||
**Fix:**
|
||||
1. Verify OmniRoute is running: `curl ${OMNIROUTE_BASE_URL}/models`
|
||||
2. Check base URL ends with `/v1`
|
||||
3. For local OmniRoute: check port (11434 vs 8000 vs other)
|
||||
|
||||
### "Unsupported provider type"
|
||||
|
||||
**Cause:** `providerType` value not in `providerTypeSchema`.
|
||||
|
||||
**Fix:** Must be one of: `anthropic`, `openai`, `google`, `openai_compatible`, `azure_openai`. For OmniRoute, use `openai_compatible`.
|
||||
|
||||
### Model returns empty or unexpected response
|
||||
|
||||
**Cause:** OmniRoute profile not available or rate-limited.
|
||||
|
||||
**Fix:**
|
||||
1. Check OmniRoute dashboard for rate limits
|
||||
2. Try a different profile (e.g., `auto` instead of `auto/coding`)
|
||||
3. Check OmniRoute status page
|
||||
|
||||
### Resolution returns wrong source
|
||||
|
||||
**Cause:** Convoy or brief override taking precedence unexpectedly.
|
||||
|
||||
**Fix:** Remember the priority order:
|
||||
1. Brief `recommended_model` (highest)
|
||||
2. Convoy `model_policy.roles[roleId]`
|
||||
3. Workspace `role_model_routes` row
|
||||
4. Pipeline defaults (lowest)
|
||||
|
||||
If you want workspace route to win, remove `model_policy` from the convoy.
|
||||
|
||||
---
|
||||
|
||||
## Security Notes
|
||||
|
||||
- **API key never in DB:** `api_key_env_var` stores the *name* of the env var, not the value
|
||||
- **API key never sent to browser:** Settings UI shows env var name only
|
||||
- **Worker reads env var at dispatch time:** Not stored in job payload
|
||||
- **Test endpoint runs server-side:** Rate-limited per workspace
|
||||
|
||||
---
|
||||
|
||||
## Related Docs
|
||||
|
||||
- **Role mapping:** `docs/omniroute-role-mapping.md` (profile → role matrix, resolution flow)
|
||||
- **ADR:** `docs/adr-model-routing.md` (architecture decision record)
|
||||
- **Skill:** `.agents/skills/model-routing/SKILL.md` (developer reference)
|
||||
- **Provider schema:** `packages/database/src/schema/model_providers.ts`
|
||||
- **Route schema:** `packages/database/src/schema/role_model_routes.ts`
|
||||
|
|
@ -18,6 +18,10 @@ export {
|
|||
type ResolveHarnessResult,
|
||||
type ResolvedHarness,
|
||||
type WorkerExecutor,
|
||||
type WorkerExecutorOptions,
|
||||
type WorkerRunResult,
|
||||
type RemoteRunOptions,
|
||||
type ResolvedModelLike,
|
||||
} from "./executor";
|
||||
|
||||
export {
|
||||
|
|
|
|||
Loading…
Reference in a new issue