ubiquitous-invention/packages/database/src/scripts/import-markdown-backlog.ts

85 lines
3 KiB
TypeScript
Raw Normal View History

/**
* One-shot markdown backlog import.
*
* Sibling of `watch-markdown-backlog.ts` but runs ONE sweep and exits. Use
* this when you want to bootstrap or refresh the DB from `plans/**` without
* leaving a long-running file watcher in place particularly important
* during agent sessions, because the watcher's upsert overwrites
* `markdown_backlog_items.status` from frontmatter on every sweep and would
* clobber `claim_task` / `complete_task` state.
*
* Until the DB markdown export side ships (see follow-up task), the
* convention is:
* - Edit `.md` files to author / refine intent.
* - Run this script when you want the changes in the DB.
* - Let agents drive runtime status via MCP tools BETWEEN syncs.
*
* Env:
* - DATABASE_URL (required)
* - MARKDOWN_BACKLOG_WORKSPACE_ID UUID of the target workspace (required)
feat(markdown-backlog): close the sync loop with DB → frontmatter export Until now the markdown importer was one-way (plans/*.md → DB). Any agent-driven status flip via claim_task / complete_task would be clobbered on the next importer sweep. This change closes the loop: the DB now projects status, priority, agent_prompt, and updated_at back into the file's frontmatter, preserving body bytes, key order, and every other frontmatter key. New: packages/database/src/markdown-backlog/export.ts - `rewriteFrontmatter()` — pure function, covered by 10 Vitest cases (round-trip identity, status flip, priority flip, agent_prompt null/block-scalar/single-line variants, body preservation, trailing-newline preservation, idempotent re-application). - `exportBacklogItemToMarkdown()` — DB-loading wrapper with atomic write (tmp + rename) and tenant fencing. Returns a structured result so callers can surface what happened in their response. Wired into: - `claim_task` MCP tool — exports on the ready → in_progress flip. - `complete_task` MCP tool — exports on any finalStatus transition. - `backlog.updateWorkflowPrompt` tRPC mutation — exports on prompt edits made through the app UI. Robust repo-root resolution (`apps/{mcp-server,web}/src/lib/repo-root.ts`, plus a copy in `import-markdown-backlog.ts`): walk up from the source file looking for `pnpm-workspace.yaml`, falling back to env var or cwd. This fixes a class of bug where `pnpm --filter <pkg>` cd's into the package directory and breaks naive cwd-based path resolution — the importer was deleting all 44 rows during smoke testing before this fix because it found zero files in `packages/database/plans/`. `config/CursorSync.md`: documents the new two-way contract, the DB-wins-on-allow-list conflict policy, and the MARKDOWN_BACKLOG_REPO_ROOT=off escape hatch for production deployments where `plans/` isn't checked out. Smoke verified end-to-end against the homelab DB: claim flips file status to in_progress, complete flips it back to ready, importer round-trips with stable content_hash (true no-op), agent identity preserved throughout. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-03 11:21:22 -04:00
* - MARKDOWN_BACKLOG_REPO_ROOT absolute path to repo root (default:
* walk up from this script looking for `pnpm-workspace.yaml`, then fall
* back to cwd). The walk is necessary because `pnpm --filter @tasks/database`
* sets cwd to `packages/database`, NOT the repo root, and a naive cwd
* resolution would scan a non-existent `packages/database/plans/`
* directory and then proceed to delete every "stale" row because
* it found zero files.
*/
feat(markdown-backlog): close the sync loop with DB → frontmatter export Until now the markdown importer was one-way (plans/*.md → DB). Any agent-driven status flip via claim_task / complete_task would be clobbered on the next importer sweep. This change closes the loop: the DB now projects status, priority, agent_prompt, and updated_at back into the file's frontmatter, preserving body bytes, key order, and every other frontmatter key. New: packages/database/src/markdown-backlog/export.ts - `rewriteFrontmatter()` — pure function, covered by 10 Vitest cases (round-trip identity, status flip, priority flip, agent_prompt null/block-scalar/single-line variants, body preservation, trailing-newline preservation, idempotent re-application). - `exportBacklogItemToMarkdown()` — DB-loading wrapper with atomic write (tmp + rename) and tenant fencing. Returns a structured result so callers can surface what happened in their response. Wired into: - `claim_task` MCP tool — exports on the ready → in_progress flip. - `complete_task` MCP tool — exports on any finalStatus transition. - `backlog.updateWorkflowPrompt` tRPC mutation — exports on prompt edits made through the app UI. Robust repo-root resolution (`apps/{mcp-server,web}/src/lib/repo-root.ts`, plus a copy in `import-markdown-backlog.ts`): walk up from the source file looking for `pnpm-workspace.yaml`, falling back to env var or cwd. This fixes a class of bug where `pnpm --filter <pkg>` cd's into the package directory and breaks naive cwd-based path resolution — the importer was deleting all 44 rows during smoke testing before this fix because it found zero files in `packages/database/plans/`. `config/CursorSync.md`: documents the new two-way contract, the DB-wins-on-allow-list conflict policy, and the MARKDOWN_BACKLOG_REPO_ROOT=off escape hatch for production deployments where `plans/` isn't checked out. Smoke verified end-to-end against the homelab DB: claim flips file status to in_progress, complete flips it back to ready, importer round-trips with stable content_hash (true no-op), agent identity preserved throughout. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-03 11:21:22 -04:00
import { existsSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { db } from "../client";
import { syncMarkdownBacklogScan } from "../markdown-backlog/sync";
feat(markdown-backlog): close the sync loop with DB → frontmatter export Until now the markdown importer was one-way (plans/*.md → DB). Any agent-driven status flip via claim_task / complete_task would be clobbered on the next importer sweep. This change closes the loop: the DB now projects status, priority, agent_prompt, and updated_at back into the file's frontmatter, preserving body bytes, key order, and every other frontmatter key. New: packages/database/src/markdown-backlog/export.ts - `rewriteFrontmatter()` — pure function, covered by 10 Vitest cases (round-trip identity, status flip, priority flip, agent_prompt null/block-scalar/single-line variants, body preservation, trailing-newline preservation, idempotent re-application). - `exportBacklogItemToMarkdown()` — DB-loading wrapper with atomic write (tmp + rename) and tenant fencing. Returns a structured result so callers can surface what happened in their response. Wired into: - `claim_task` MCP tool — exports on the ready → in_progress flip. - `complete_task` MCP tool — exports on any finalStatus transition. - `backlog.updateWorkflowPrompt` tRPC mutation — exports on prompt edits made through the app UI. Robust repo-root resolution (`apps/{mcp-server,web}/src/lib/repo-root.ts`, plus a copy in `import-markdown-backlog.ts`): walk up from the source file looking for `pnpm-workspace.yaml`, falling back to env var or cwd. This fixes a class of bug where `pnpm --filter <pkg>` cd's into the package directory and breaks naive cwd-based path resolution — the importer was deleting all 44 rows during smoke testing before this fix because it found zero files in `packages/database/plans/`. `config/CursorSync.md`: documents the new two-way contract, the DB-wins-on-allow-list conflict policy, and the MARKDOWN_BACKLOG_REPO_ROOT=off escape hatch for production deployments where `plans/` isn't checked out. Smoke verified end-to-end against the homelab DB: claim flips file status to in_progress, complete flips it back to ready, importer round-trips with stable content_hash (true no-op), agent identity preserved throughout. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-03 11:21:22 -04:00
function findRepoRootByMarker(start: string): string | null {
let dir = start;
for (let i = 0; i < 10; i++) {
if (existsSync(join(dir, "pnpm-workspace.yaml"))) return dir;
const parent = dirname(dir);
if (parent === dir) return null;
dir = parent;
}
return null;
}
function resolveRepoRoot(): string {
const envOverride = process.env.MARKDOWN_BACKLOG_REPO_ROOT?.trim();
if (envOverride && envOverride.length > 0) return resolve(envOverride);
const fileDir = dirname(fileURLToPath(import.meta.url));
const fromFile = findRepoRootByMarker(fileDir);
if (fromFile) return fromFile;
const fromCwd = findRepoRootByMarker(process.cwd());
if (fromCwd) return fromCwd;
return resolve(process.cwd());
}
const workspaceId = process.env.MARKDOWN_BACKLOG_WORKSPACE_ID?.trim();
feat(markdown-backlog): close the sync loop with DB → frontmatter export Until now the markdown importer was one-way (plans/*.md → DB). Any agent-driven status flip via claim_task / complete_task would be clobbered on the next importer sweep. This change closes the loop: the DB now projects status, priority, agent_prompt, and updated_at back into the file's frontmatter, preserving body bytes, key order, and every other frontmatter key. New: packages/database/src/markdown-backlog/export.ts - `rewriteFrontmatter()` — pure function, covered by 10 Vitest cases (round-trip identity, status flip, priority flip, agent_prompt null/block-scalar/single-line variants, body preservation, trailing-newline preservation, idempotent re-application). - `exportBacklogItemToMarkdown()` — DB-loading wrapper with atomic write (tmp + rename) and tenant fencing. Returns a structured result so callers can surface what happened in their response. Wired into: - `claim_task` MCP tool — exports on the ready → in_progress flip. - `complete_task` MCP tool — exports on any finalStatus transition. - `backlog.updateWorkflowPrompt` tRPC mutation — exports on prompt edits made through the app UI. Robust repo-root resolution (`apps/{mcp-server,web}/src/lib/repo-root.ts`, plus a copy in `import-markdown-backlog.ts`): walk up from the source file looking for `pnpm-workspace.yaml`, falling back to env var or cwd. This fixes a class of bug where `pnpm --filter <pkg>` cd's into the package directory and breaks naive cwd-based path resolution — the importer was deleting all 44 rows during smoke testing before this fix because it found zero files in `packages/database/plans/`. `config/CursorSync.md`: documents the new two-way contract, the DB-wins-on-allow-list conflict policy, and the MARKDOWN_BACKLOG_REPO_ROOT=off escape hatch for production deployments where `plans/` isn't checked out. Smoke verified end-to-end against the homelab DB: claim flips file status to in_progress, complete flips it back to ready, importer round-trips with stable content_hash (true no-op), agent identity preserved throughout. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-03 11:21:22 -04:00
const repoRootAbs = resolveRepoRoot();
if (!workspaceId) {
console.error(
"[import-markdown-backlog] Set MARKDOWN_BACKLOG_WORKSPACE_ID to your workspace UUID.",
);
process.exit(1);
}
void (async () => {
try {
const result = await syncMarkdownBacklogScan(db, {
workspaceId,
repoRootAbs,
});
console.log(
`[import-markdown-backlog] done — scanned ${result.scannedFiles} files, upserted ${result.upsertedRows} rows, deleted ${result.deletedRows} stale rows. Repo root: ${repoRootAbs}`,
);
process.exit(0);
} catch (e) {
console.error("[import-markdown-backlog] sync failed:", e);
process.exit(1);
}
})();