67 lines
2 KiB
Markdown
67 lines
2 KiB
Markdown
|
---
|
||
|
|
kind: task
|
||
|
|
slug: worker-checkpoint-resume
|
||
|
|
title: Worker checkpoint and crash recovery
|
||
|
|
plan_slug: automation-foundation
|
||
|
|
epic_slug: apps-worker-v1
|
||
|
|
status: draft
|
||
|
|
priority: P1
|
||
|
|
tenant_id: global
|
||
|
|
owner: unassigned
|
||
|
|
cursor_todo_id: null
|
||
|
|
updated_at: "2026-08-03"
|
||
|
|
---
|
||
|
|
|
||
|
|
# Task summary
|
||
|
|
|
||
|
|
Persist worker session progress at configurable checkpoints so a crashed or stalled job resumes from the last checkpoint instead of restarting the brief from scratch.
|
||
|
|
|
||
|
|
## Description
|
||
|
|
|
||
|
|
**Benchmark:** [Autensa mission-control](https://github.com/crshdn/mission-control) checkpoint & crash recovery — study their checkpoint restore API and per-task history UX; implement Echodo-native equivalents on Postgres.
|
||
|
|
|
||
|
|
### Schema
|
||
|
|
|
||
|
|
Add `worker_checkpoints` (or JSONB on `agent_runs`) workspace-scoped:
|
||
|
|
|
||
|
|
- `agent_run_id`, `convoy_id`, `brief_id`
|
||
|
|
- `checkpoint_seq`, `tool_history` (bounded), `files_touched`, `branch`, `worktree_path`
|
||
|
|
- `occurred_at`
|
||
|
|
|
||
|
|
### Worker behavior
|
||
|
|
|
||
|
|
- Save checkpoint after each tool round (configurable via `WORKER_CHECKPOINT_EVERY_N_STEPS`, default 1).
|
||
|
|
- On job start: if open run has latest checkpoint, resume tool loop from saved state.
|
||
|
|
- Orchestrator stall detector can trigger re-dispatch with `resume_from_checkpoint: true`.
|
||
|
|
|
||
|
|
### Operator surface (minimal v1)
|
||
|
|
|
||
|
|
- Mission control detail panel: checkpoint count + last checkpoint time (full history deferred).
|
||
|
|
|
||
|
|
## Dependencies
|
||
|
|
|
||
|
|
- [Task-worker-minimal-tool-loop](./Task-worker-minimal-tool-loop.md) (baseline loop must exist).
|
||
|
|
|
||
|
|
## Subtasks
|
||
|
|
|
||
|
|
- [ ] Zod schemas in `packages/shared/worker`
|
||
|
|
- [ ] Drizzle migration + workspace-scoped queries
|
||
|
|
- [ ] Checkpoint write in `minimal-session` / `handle-job`
|
||
|
|
- [ ] Resume path on re-dispatch
|
||
|
|
- [ ] Env docs in `docs/worker.md`
|
||
|
|
|
||
|
|
## Estimation
|
||
|
|
|
||
|
|
M
|
||
|
|
|
||
|
|
## Acceptance criteria
|
||
|
|
|
||
|
|
- [ ] Kill worker mid-brief; re-dispatch resumes from last tool step without re-cloning
|
||
|
|
- [ ] Checkpoints filtered by `workspace_id`
|
||
|
|
- [ ] Checkpoint payload does not store API keys or full file contents (paths + hashes only)
|
||
|
|
|
||
|
|
## Links
|
||
|
|
|
||
|
|
- Epic: `./Epic-apps-worker-v1.md`
|
||
|
|
- Blocks: reliable unattended dogfood for Epic 8
|