65 lines
2.1 KiB
Markdown
65 lines
2.1 KiB
Markdown
|
---
|
||
|
|
kind: task
|
||
|
|
slug: orchestrator-auto-nudge
|
||
|
|
title: Orchestrator agent health — auto-nudge and zombie recovery
|
||
|
|
plan_slug: automation-foundation
|
||
|
|
epic_slug: apps-orchestrator
|
||
|
|
status: draft
|
||
|
|
priority: P2
|
||
|
|
tenant_id: global
|
||
|
|
owner: unassigned
|
||
|
|
cursor_todo_id: null
|
||
|
|
updated_at: "2026-08-03"
|
||
|
|
---
|
||
|
|
|
||
|
|
# Task summary
|
||
|
|
|
||
|
|
Extend stall detection beyond marking convoys `blocked`: detect zombie agent runs, optionally re-dispatch or nudge, and emit health events for mission control / fleet feed.
|
||
|
|
|
||
|
|
## Description
|
||
|
|
|
||
|
|
**Benchmark:** Autensa agent health monitoring with auto-nudge and convoy health panel. Echodo already has `ORCHESTRATOR_STALL_MINUTES` and `detectStalledConvoys` — this task adds **recovery actions** and observability.
|
||
|
|
|
||
|
|
### Health states
|
||
|
|
|
||
|
|
| State | Trigger | Default action |
|
||
|
|
|-------|---------|----------------|
|
||
|
|
| **Stalled** | No heartbeat on open `agent_run` for N minutes | Mark convoy `blocked`; surface in gate inbox sidebar |
|
||
|
|
| **Zombie** | Run open but worker process gone | Re-enqueue job with `resume_from_checkpoint` when Epic 7 checkpoint task ships |
|
||
|
|
| **Nudge** | Soft stall (configurable) | Log `convoy_events` health event; optional Redis ping to worker |
|
||
|
|
|
||
|
|
### Config (env)
|
||
|
|
|
||
|
|
- `ORCHESTRATOR_NUDGE_MINUTES` (optional, < stall threshold)
|
||
|
|
- `ORCHESTRATOR_AUTO_REDISPATCH` (default false — opt-in for unattended dogfood)
|
||
|
|
|
||
|
|
### Events
|
||
|
|
|
||
|
|
Append `convoy_events` with `event_type: health` and metadata `{ reason, action, agent_run_id }` for activity feed (Epic 9).
|
||
|
|
|
||
|
|
## Dependencies
|
||
|
|
|
||
|
|
- Epic 6 orchestrator service (shipped)
|
||
|
|
- [Task-worker-checkpoint-resume](../Epic-apps-worker-v1/Task-worker-checkpoint-resume.md) for meaningful re-dispatch
|
||
|
|
|
||
|
|
## Subtasks
|
||
|
|
|
||
|
|
- [ ] Extend `detectStalledConvoys` with nudge vs hard-stall tiers
|
||
|
|
- [ ] Optional re-dispatch path on zombie detection
|
||
|
|
- [ ] `convoy_events` health event writer
|
||
|
|
- [ ] Document in `docs/orchestrator.md`
|
||
|
|
|
||
|
|
## Estimation
|
||
|
|
|
||
|
|
M
|
||
|
|
|
||
|
|
## Acceptance criteria
|
||
|
|
|
||
|
|
- [ ] Stalled run moves convoy to `blocked` and writes health event
|
||
|
|
- [ ] With `ORCHESTRATOR_AUTO_REDISPATCH=1`, zombie run triggers new job enqueue
|
||
|
|
- [ ] No unbounded re-dispatch loops (max retries per brief configurable)
|
||
|
|
|
||
|
|
## Links
|
||
|
|
|
||
|
|
- Epic: `./Epic-apps-orchestrator.md`
|