9eef8d9575
The pipeline metrics shim worked correctly through Jun 4 (65 role events, 25 convoys recorded) but silently stopped capturing thereafter. 8 convoy PRs merged Jun 5-11 (#126-#133) with zero rows logged to .convoys/.metrics.jsonl. The roles' Metrics sections clearly instruct the agents to call scripts/log-convoy-event.sh after every hand-off, but the instruction was skipped during multitask audit fan-outs and longer sessions where the Metrics section fell out of working context. Two changes to prevent the silent gap from recurring: 1. .gitignore: drop the `.convoys/.metrics.jsonl` ignore line. Convoy telemetry now committed in git so gaps surface in PR review. The script comment was already clear that events contain metadata only — no code, no prompts. 2. .github/workflows/convoy-metrics-gate.yml: new CI gate that fails any PR titled `convoy:` if no rows were added to .convoys/.metrics.jsonl between base and head. Bypass with the `skip-metrics` label + a documented reason. Non-convoy PRs are no-op. Also: commits the existing 65-event history to git so future analysis (and the §10 measurement protocol in agent-pipeline's v0.4 plan) has a stable baseline to compare against. Runs on self-hosted axiom runner to inherit the lower GH minutes cost the Jun 5-11 work already migrated to (PR #132). Co-authored-by: Cursor <cursoragent@cursor.com>
90 lines
3.8 KiB
YAML
90 lines
3.8 KiB
YAML
name: Convoy metrics gate
|
|
|
|
# Forces convoy PRs to include role-event telemetry. Without this gate, the
|
|
# pipeline produces work but no signal — exactly what happened during the
|
|
# Jun 5-11 experiment window when 8 convoy PRs shipped without logging a
|
|
# single .metrics.jsonl row (see analytics/v0.4-beta1-results.md in
|
|
# agent-pipeline).
|
|
#
|
|
# Triggers: PR titled `convoy:` (case-insensitive prefix). If a convoy PR
|
|
# does NOT add at least one new role-event line to .convoys/.metrics.jsonl,
|
|
# this check fails. Bypass: add the `skip-metrics` label to the PR (rare,
|
|
# document the reason in the PR body).
|
|
#
|
|
# Other PRs are no-op (skip the gate).
|
|
|
|
on:
|
|
pull_request:
|
|
branches: [main]
|
|
types: [opened, synchronize, reopened, edited, labeled, unlabeled]
|
|
|
|
permissions:
|
|
contents: read
|
|
pull-requests: read
|
|
|
|
jobs:
|
|
gate:
|
|
name: Require role-event telemetry on convoy PRs
|
|
runs-on: [self-hosted, axiom]
|
|
steps:
|
|
- name: Inspect PR
|
|
id: inspect
|
|
uses: actions/github-script@v7
|
|
with:
|
|
script: |
|
|
const pr = context.payload.pull_request;
|
|
const title = pr.title || '';
|
|
const labels = (pr.labels || []).map(l => l.name);
|
|
const isConvoy = /^convoy:/i.test(title);
|
|
const hasBypass = labels.includes('skip-metrics');
|
|
core.setOutput('is_convoy', isConvoy ? 'true' : 'false');
|
|
core.setOutput('has_bypass', hasBypass ? 'true' : 'false');
|
|
core.info(`title="${title}" is_convoy=${isConvoy} has_bypass=${hasBypass}`);
|
|
if (hasBypass) {
|
|
core.warning(`skip-metrics label present on PR #${pr.number} — gate will exit early. Document the reason in the PR body.`);
|
|
}
|
|
|
|
- name: Skip — not a convoy PR
|
|
if: steps.inspect.outputs.is_convoy != 'true'
|
|
run: |
|
|
echo "PR title is not 'convoy:' — gate is no-op for this PR."
|
|
|
|
- name: Skip — bypass label present
|
|
if: steps.inspect.outputs.is_convoy == 'true' && steps.inspect.outputs.has_bypass == 'true'
|
|
run: |
|
|
echo "::warning::skip-metrics label bypass. Metrics gate not enforced for this PR."
|
|
|
|
- name: Checkout PR head
|
|
if: steps.inspect.outputs.is_convoy == 'true' && steps.inspect.outputs.has_bypass != 'true'
|
|
uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 0
|
|
|
|
- name: Verify metrics rows added
|
|
if: steps.inspect.outputs.is_convoy == 'true' && steps.inspect.outputs.has_bypass != 'true'
|
|
run: |
|
|
set -euo pipefail
|
|
|
|
BASE_SHA="${{ github.event.pull_request.base.sha }}"
|
|
HEAD_SHA="${{ github.event.pull_request.head.sha }}"
|
|
METRICS=".convoys/.metrics.jsonl"
|
|
|
|
if [ ! -f "$METRICS" ]; then
|
|
echo "::error::Convoy PR but $METRICS does not exist. The L2 roles must call scripts/log-convoy-event.sh after each hand-off — see .cursor/agents/role-*.md 'Metrics' sections."
|
|
exit 1
|
|
fi
|
|
|
|
ADDED_LINES=$(git diff "$BASE_SHA"..."$HEAD_SHA" -- "$METRICS" | grep -c '^+{' || true)
|
|
|
|
if [ "$ADDED_LINES" -eq 0 ]; then
|
|
echo "::error::Convoy PR did not add any rows to $METRICS."
|
|
echo "::error::Did the L2 roles call scripts/log-convoy-event.sh after their hand-off? See .cursor/agents/role-*.md 'Metrics' section."
|
|
echo "::error::To bypass intentionally (rare), add 'skip-metrics' label to the PR and document why in the PR body."
|
|
exit 1
|
|
fi
|
|
|
|
if ! git diff "$BASE_SHA"..."$HEAD_SHA" -- "$METRICS" | grep -q '"convoy"'; then
|
|
echo "::warning::Added lines do not appear to be well-formed JSON with a 'convoy' field. Verify scripts/log-convoy-event.sh ran cleanly."
|
|
fi
|
|
|
|
echo "::notice::Convoy metrics gate passed: $ADDED_LINES new role-event row(s) added to $METRICS."
|