Submatter/plans/Plan-automation-foundation/Epic-repo-discovery-onboarding/Task-repo-doc-ingestion.md

kind	slug	title	plan_slug	epic_slug	status	priority	tenant_id	owner	cursor_todo_id	updated_at
task	repo-doc-ingestion	Idempotent repository context ingestion	automation-foundation	repo-discovery-onboarding	ready	P1	global	unassigned	null	2026-09-25

Task summary

Provide an explicit, retryable import that reads bounded repository documentation and existing convoy manifests through the forge API, stores source provenance, and avoids duplicate Echodo objects.

Description

Remote reads should not lengthen or roll back the batch-connect transaction. After connection, an admin starts import and sees per-repo status. V1 scans only root README.md, AGENTS.md, CONTRIBUTING.md, and .convoys/*.md; recursive docs/** and plans/** import is deferred.

Subtasks

• Add bounded file/list-directory forge contracts with response, media type, file-count, and byte-size validation.
• Extract pure convoy markdown parsing into a neutral shared package so web and MCP do not import each other's app source.
• Define durable source identity and import status by workspace, connected repo, and source path; generate a new migration if schema changes are needed.
• Store docs as the canonical document object type with forge/repo/path/revision metadata.
• Reconcile valid .convoys/*.md into workspace-scoped projects and valid brief metadata into child tasks.
• Add admin-only import start/status/retry APIs or an equivalent bounded background execution mechanism.
• Report imported, unchanged, skipped, and failed counts without aborting successful files.
• Test idempotency, malformed manifests, missing optional files, partial failure, limits, and tenant isolation.

Security and sync constraints

• Remote files are read-only in this task.
• Re-import updates only objects with matching source identity and a changed revision.
• Missing files are skipped, while authentication/rate-limit/parse/persistence failures are visible.
• Read config/CursorSync.md before changing reconciliation behavior.

Estimation

L

Acceptance criteria

• Re-running an unchanged import creates no duplicate documents, convoys, or briefs.
• Imported docs use type: "document" and record source repo/path/revision.
• Valid convoy manifests appear in Mission Control; malformed manifests do not block other imports.
• File count and byte limits prevent unbounded repository ingestion.
• Failures are retryable and reported per repo/file.
• New schema changes include generated migration SQL.
• Lint, type-check, test, and relevant builds pass.

Links

• Epic: ./Epic-repo-discovery-onboarding.md
• Depends on: ./Task-forge-discovery-api.md, ./Task-repo-picker-modal.md
• Convoy brief: ../../../.convoys/repo-discovery-onboarding/brief-3-ingestion.md