rstillwell/Submatter
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions
Submatter/.convoys/repo-discovery-onboarding/brief-3-ingestion.md

3.3 KiB
Raw Blame History

brief convoy role title status priority slice_dependencies recommended_model model_tier workspace_id
3 repo-discovery-onboarding role-implementer Idempotent repository context ingestion ready P1
1
2
qwen-coder local global

Brief 3 — Idempotent repository context ingestion

Goal

After repositories are connected, an explicit retryable ingestion action imports supported repository documentation and existing convoy manifests with provenance, without making remote API latency part of the batch-connect transaction.

Backlog task: plans/Plan-automation-foundation/Epic-repo-discovery-onboarding/Task-repo-doc-ingestion.md.

Architecture constraints

  • Use the canonical object type document, not doc.
  • Do not import an unbounded repository tree. V1 paths are root README.md, AGENTS.md, CONTRIBUTING.md, and .convoys/*.md; recursive docs/** and plans/** are deferred until limits and conflict semantics are designed.
  • Import is idempotent by (workspace, connectedRepoId, sourcePath), and records source forge, repo, path, revision/sha, and last import result in metadata.
  • Keep remote content read-only. Re-import updates the matching imported object only when the remote revision changes.
  • Share convoy parsing/reconciliation logic from a neutral package; do not import apps/mcp-server source into apps/web.
  • A missing optional file is not an error. Authentication, rate-limit, parse, and persistence failures are visible per repo and retryable.

Files

  • Shared connected-repo schemas/services — add bounded tree/list-file and fetch-text-file contracts with size/type validation.
  • Forge adapters — implement GitHub and Forgejo/Gitea directory/file reads against the connected repo's default branch.
  • Extract reusable convoy markdown parsing from apps/mcp-server/src/lib/convoy-files.ts into a package usable by web and MCP, preserving local-file behavior.
  • Database schema/migration if needed — add durable repository-source identity and ingestion status. Every new tenant row includes workspace_id and an index.
  • Web tRPC/service — add admin-only startImport/importStatus or an equivalent bounded background execution path. Do not perform all remote reads inside the batchConnect transaction.

Acceptance

  • Importing a repo creates or updates document objects for supported files with source provenance.
  • Re-running an unchanged import creates no duplicates.
  • Existing .convoys/*.md manifests reconcile to workspace-scoped convoy projects. Brief files are imported only when their schema is valid and can be linked safely.
  • One bad file does not roll back successful files; the result reports imported, unchanged, skipped, and failed counts.
  • File count and byte limits prevent accidental large imports.
  • Unit/integration tests cover idempotency, missing files, malformed convoy frontmatter, tenant isolation, and partial failure.
  • Generate and commit a new migration for schema changes; lint, type-check, test, and relevant build checks pass.

Prompt for implementer

Implement only Brief 3 after Briefs 1 and 2. Read config/CursorSync.md before changing sync behavior. Preserve the local-first .convoys/ contract and never mutate remote files. Keep ingestion bounded and retryable; do not silently invent conflict resolution for plans/**.