ubiquitous-invention/plans/Plan-agent-coordination/Epic-task-as-runnable-unit/Task-export-db-to-markdown-frontmatter.md
Randall Stillwell 56b697b81c feat(markdown-backlog): close the sync loop with DB → frontmatter export
Until now the markdown importer was one-way (plans/*.md → DB). Any
agent-driven status flip via claim_task / complete_task would be
clobbered on the next importer sweep. This change closes the loop: the
DB now projects status, priority, agent_prompt, and updated_at back
into the file's frontmatter, preserving body bytes, key order, and
every other frontmatter key.

New: packages/database/src/markdown-backlog/export.ts
  - `rewriteFrontmatter()` — pure function, covered by 10 Vitest cases
    (round-trip identity, status flip, priority flip, agent_prompt
    null/block-scalar/single-line variants, body preservation,
    trailing-newline preservation, idempotent re-application).
  - `exportBacklogItemToMarkdown()` — DB-loading wrapper with atomic
    write (tmp + rename) and tenant fencing. Returns a structured
    result so callers can surface what happened in their response.

Wired into:
  - `claim_task` MCP tool — exports on the ready → in_progress flip.
  - `complete_task` MCP tool — exports on any finalStatus transition.
  - `backlog.updateWorkflowPrompt` tRPC mutation — exports on prompt
    edits made through the app UI.

Robust repo-root resolution (`apps/{mcp-server,web}/src/lib/repo-root.ts`,
plus a copy in `import-markdown-backlog.ts`): walk up from the source
file looking for `pnpm-workspace.yaml`, falling back to env var or cwd.
This fixes a class of bug where `pnpm --filter <pkg>` cd's into the
package directory and breaks naive cwd-based path resolution — the
importer was deleting all 44 rows during smoke testing before this fix
because it found zero files in `packages/database/plans/`.

`config/CursorSync.md`: documents the new two-way contract, the
DB-wins-on-allow-list conflict policy, and the
MARKDOWN_BACKLOG_REPO_ROOT=off escape hatch for production deployments
where `plans/` isn't checked out.

Smoke verified end-to-end against the homelab DB: claim flips file
status to in_progress, complete flips it back to ready, importer
round-trips with stable content_hash (true no-op), agent identity
preserved throughout.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-03 10:21:22 -05:00

5 KiB

kind slug title plan_slug epic_slug status priority tenant_id owner cursor_todo_id updated_at
task export-db-to-markdown-frontmatter Export DB state back to markdown frontmatter (close the sync loop) agent-coordination task-as-runnable-unit done P1 global unassigned null 2026-06-03

Task summary

The markdown importer at packages/database/src/markdown-backlog/sync.ts is one-way: it reads plans/**/*.md and upserts into markdown_backlog_items. There is no path back from the DB to the markdown frontmatter. As soon as an agent calls complete_task (status → done), running the importer would clobber that transition back to whatever the markdown still says.

For now we paper over this by running the importer once at bootstrap and ad-hoc thereafter (pnpm --filter @tasks/database import:markdown-backlog). That's tolerable for a single-operator dogfood loop, but it doesn't scale and it's a correctness bug: any change made through the app — status flips, prompt edits, new tasks created in the UI — is at risk of being silently reverted.

This task closes that loop by adding a DB → markdown export.

Description

Scope

Implement exportBacklogItemToMarkdown(db, { workspaceId, backlogItemId }) that:

  1. Loads the markdown_backlog_items row.
  2. Locates the corresponding .md file on disk using repo_relative_path.
  3. Reads the existing file (must preserve the body verbatim).
  4. Rewrites the YAML frontmatter so the canonical fields match the DB row:
    • status
    • priority
    • title (only when changed; titles are usually human-authored)
    • updated_at (bumped to today on any rewrite)
    • workflow_prompt → frontmatter agent_prompt: (omit if null; serialize as a block scalar | if multi-line)
  5. Preserves every other frontmatter key untouched, in the original order.
  6. Writes the file back atomically (write to tmp, rename) to avoid half-written reads from any watcher.

Wire it into:

  • complete_task (after the status transition commits): export the touched row.
  • claim_task: optional — probably yes when status flipped to in_progress.
  • backlog.updateWorkflowPrompt tRPC mutation (apps/web/server/routers/backlog.ts).
  • Maybe workspaces.archive cascade for backlog items (lower priority — archive is a UI-driven verb and the markdown may not need to reflect it).

Out of scope:

  • Creating brand-new .md files from DB rows (we never UI-create backlog items today; punt to a separate task).
  • Two-way conflict resolution. If someone edits both the file and the DB row between importer runs we just take the DB. Document this clearly in config/CursorSync.md.

Implementation notes

  • Use a small, well-tested YAML rewriter, not a regex. The yaml package (already a dep of @tasks/database) supports preserving comments and key order via Document parsing. Use Document.parse(source), mutate scalar values in place, and String(doc) to serialize.
  • File I/O lives in packages/database/src/markdown-backlog/export.ts. Keep this module free of tRPC / Next imports so the MCP server can call it directly.
  • Add Vitest cases covering:
    • Round-trip: read → no change → write produces identical bytes.
    • Status update: readydone only mutates status and updated_at.
    • Setting agent_prompt: null removes the key (does not write agent_prompt: null).
    • Block-scalar serialization for multi-line prompts.
    • Preserves trailing newline and body separator (---\n\n vs ---\n).
  • Behind a workspace-level setting markdown_export_enabled (default true). Some operators may not want the app rewriting their plans/ tree; let them opt out.

Acceptance

  1. After calling complete_task with finalStatus: done, the corresponding .md file's frontmatter status reads done on disk.
  2. Running pnpm --filter @tasks/database import:markdown-backlog immediately after produces zero new upserts (the file already matches the DB).
  3. Vitest coverage of the export helper.
  4. config/CursorSync.md is updated to describe the new two-way contract and the conflict policy (DB wins on conflict between importer runs).
  5. No diff noise: re-exporting an unchanged row leaves the file byte-identical.

Risks

  • YAML formatting is finicky; agents may produce huge accidental diffs the first time this runs. Mitigate with the round-trip test and an --dry-run flag on the export helper for first verification.
  • Concurrent writes: an operator editing the .md in their editor at the moment of export could lose changes. The atomic write helps but doesn't fully solve it. Document the contract: while agents are running, treat the .md files as derived state.