Benchmarks

kanros is built on the premise that the eval harness itself should be invisible: when you run a suite, the time and memory you spend should go to your models, not to the orchestrator. This page documents how we measure that overhead and the numbers the current release posts.

All numbers below are medians over 10 iterations on an Apple-silicon macOS laptop, measured with GNU time (gtime) against release builds of kanros v0.1.0-rc.1.

Headline numbers

The benchmark workload is a 30-cell eval — 1 prompt × 30 tests × 1 provider, four assertions per test (contains, regex, length, not-contains) — run with --no-cache so nothing is served from the response cache:

The eval uses the built-in echo provider, which is deterministic and fully offline — so wall-clock and RSS reflect pure orchestrator overhead: config parsing, matrix expansion, the concurrency pool, assertion evaluation, and report serialization. No network or model latency is hiding in the numbers.

Why this matters

• CI turnaround. A harness that adds seconds of fixed overhead per invocation taxes every PR. kanros' overhead is small enough that eval wall-clock is effectively your providers' latency and nothing else.
• Memory headroom. ~15 MiB peak RSS means you can run kanros beside a local model server, inside a small CI runner, or many instances in parallel without budgeting memory for the harness.
• Cold start. The static binary loads in under the 10 ms resolution of GNU time. There is no runtime to boot, no module graph to resolve, no JIT warm-up — which matters when a pipeline shells out to the tool many times.

Comparison baseline

To keep ourselves honest, the same workload is also maintained for a widely used Node.js-based eval CLI, with the identical 30-test matrix, the same echo-style offline provider, and equivalent assertions (the one assertion the other tool lacks as a built-in is expressed there as an equivalent JavaScript expression). Same machine, same methodology, medians over the same number of iterations:

Two caveats we apply to our own numbers:

• The wall-clock factor is flattered by the Node.js runtime's startup cost. Measuring warm iterations only, kanros remains roughly ~50x faster on this workload — that is the fairer steady-state figure.
• This workload measures the harness, not the models. On a real suite dominated by provider latency, both tools spend most of their wall-clock waiting on the same API responses; what differs is the fixed overhead added on top and the memory held while waiting.

Methodology

1. Offline, deterministic provider. The echo provider removes network and model variance entirely. What remains is the orchestrator.
2. --no-cache on every run. Response-cache hits would measure SQLite lookup speed, not eval speed.
3. Cheapest output format (--format json), matching the steady-state CI usage pattern.
4. Peak RSS via GNU time — maximum resident set size observed by the kernel for the whole process tree (gtime -v on macOS, /usr/bin/time -v on Linux).
5. Medians, not means. First iterations of JIT-based runtimes are dominated by warm-up; the median windows that out on both sides rather than penalising either tool for its first run.
6. Release gate. These bars are enforced as part of the release process — a release candidate that regresses past the target ratios does not ship.

Microbenchmarks

Beyond the end-to-end harness numbers, kanros-bench maintains Criterion microbenchmarks for the runner hot paths (matrix expansion, assertion evaluation, cache lookups, report serialization). Cold start is also timed there directly in-process, without the 10 ms child-process measurement floor.

Reproducing

The benchmark harness ships with the source distribution and writes a machine-readable results.json (medians, ratios, pass/fail verdicts). Numbers are refreshed on every release candidate; the committed results above are from 2026-05-30. Expect Linux runs to widen the margins slightly — Node.js starts slower on typical Linux CI hosts, while kanros' static binary does not care.

If you benchmark kanros yourself and see materially different numbers, please email support@kanros.dev with your results.json — we treat performance regressions as bugs.