kanros diff

Compare two completed runs and report which cells changed. Useful for:

Telling whether a prompt edit regressed an eval suite.
Surfacing flaky tests across nightly runs.
Building a CI gate that fails the PR if specific cells flipped from pass to fail.

kanros diff <RUN_A> <RUN_B> [--format <FMT>] [--verbose] [--cache-path <PATH>]

Flag / arg	Default	Meaning
`<RUN_A>`	required	Baseline run identifier.
`<RUN_B>`	required	Comparison run identifier.
`--format <FMT>`	`terminal`	Output format. `terminal` or `json`.
`-v, --verbose`	off	Show stable (unchanged) cells in terminal output.
`--cache-path <PATH>`	XDG cache dir	Custom history database location.

Run identifiers

A run identifier is one of:

A run UUID (the value kanros run prints in its header).
latest — the most recent run.
latest~N — the run made N runs before the most recent (latest~1, latest~2, …).

The history is persisted automatically: kanros run writes the run summary to the cache database at the end of every invocation. This happens even if you passed --no-cache (response caching is separate from history).

Examples

kanros diff latest~1 latest
kanros diff 8c2c5c9e-… b0e21d1c-…
kanros diff latest~1 latest --format json > diff.json
kanros diff latest~1 latest --verbose

What the diff shows

Each cell is identified by (provider_id, prompt_index, test_index). The diff classifies each cell as one of:

Status	Meaning
`stable`	Same outcome in both runs. Hidden by default; shown with `--verbose`.
`regression`	`pass` in `RUN_A`, `fail`/`error` in `RUN_B`. This is what CI wants to gate.
`recovery`	`fail`/`error` in `RUN_A`, `pass` in `RUN_B`.
`added`	Cell present in `RUN_B` only (test matrix grew).
`removed`	Cell present in `RUN_A` only.

The terminal output groups cells by status:

kanros diff 8c2c5c9e… vs b0e21d1c…
  3 regressions, 1 recovery, 0 added, 0 removed, 41 stable

  [REGRESSION] openai:gpt-4o-mini   prompt#0  test#5   contains failed: "key findings"
  [REGRESSION] openai:gpt-4o-mini   prompt#1  test#0   latency failed: 6120ms > 5000ms
  [REGRESSION] anthropic:claude-3-5-haiku  prompt#0  test#7   llm-rubric failed: …
  [RECOVERY]   openai:gpt-4o-mini   prompt#0  test#3   …

The JSON output is a single object with regressions, recoveries, added, removed, and stable arrays. Each entry includes the cell coordinates, the RUN_A and RUN_B outcomes, and the offending assertion (if any).

CI integration

A simple gate that fails when regressions exist:

kanros run --config kanros.yaml
kanros diff latest~1 latest --format json > diff.json
if [ "$(jq '.regressions | length' diff.json)" -gt 0 ]; then
  echo "regressions detected:"
  jq -r '.regressions[] | "\(.provider_id) prompt#\(.prompt_index) test#\(.test_index): \(.reason)"' diff.json
  exit 1
fi

For GitHub Actions, prefer --format github on the underlying run and let the diff feed a follow-up step that posts a PR comment.

Custom cache path

When CI uses a per-PR cache database, point diff at the same file:

kanros run --cache-path ./.kanros-cache.db
kanros diff --cache-path ./.kanros-cache.db latest~1 latest

If you forget the --cache-path, diff looks in the default platform cache directory and may not find the run.

What the diff cannot see

Provider drift inside a stable cell. If the response changed but the assertion still passes, the cell is stable. The cache stores the latest response per (provider_id, request) so older responses are not retained for byte-level comparison.
Timing jitter. Latency assertions diff their pass/fail outcome, but the raw latency numbers are only surfaced in --format json.
Streaming traces. Mid-stream snapshots are not part of the run summary.

Notes

latest~N is resolved against the run history at command time. If you call diff from a script that just ran an eval, latest is the run you just produced, and latest~1 is the previous one.
The history is small (one row per run) and not pruned automatically. If it grows unwieldy you can wipe it via kanros cache clear — note that this also clears the response cache.