kanros run

kanros run executes an evaluation suite. It is the workhorse of the CLI; every other subcommand exists either to feed run (init, migrate), to inspect what it produced (diff, share, cache), or to handle the red-team analogue (redteam run).

Usage

kanros run [OPTIONS]

Options

Flag	Default	Meaning
`-c, --config <PATH>`	`kanros.yaml`	Config file (YAML / TOML / JSON).
`--format <FMT>`	`terminal`	Output format. See list below.
`--no-cache`	off	Disable the response cache for this run.
`--cache-path <PATH>`	XDG cache dir	Custom cache database file.
`--plugins`	off	Enable WASM plugin support (`type: custom` and `type: wasm`). Without it those assertions error.
`--concurrency <N>`	`runner.max_concurrency` (32)	Max provider calls in flight. Overrides the config; `1` runs cells one at a time. Also honoured with `--watch`.
`--max-cost <USD>`	`runner.max_cost_usd`	Stop scheduling new cells once estimated spend exceeds this.
`--watch`	off	Re-run on every change to the config or referenced prompt files.
`--output-format promptfoo`	unset	Emit promptfoo's output JSON shape (takes precedence over `--format`).
`--webui <PORT>`	unset	Serve a live web UI on `127.0.0.1:<PORT>` while the run executes.
`--stream`	off	Stream the first cell's completion to stderr before the eval.
`--batch`	off	Submit the matrix through the provider's batch API (OpenAI).
`--batch-poll-interval <SECS>`	`30`	Poll interval for `--batch`.
`--github-check <OWNER/REPO>`	unset	Publish the result as a GitHub check run (needs `GITHUB_TOKEN`).
`--github-sha <SHA>`	`$GITHUB_SHA`	Commit for `--github-check`.

Global flags --quiet, --verbose, and --json-logs apply to all subcommands.

Output formats

Format	Description
`terminal`	Human-readable table grouped by cell.
`json`	Pretty JSON; the full `RunSummary` struct plus top-level `pass_count`, `fail_count`, `error_count` and `total_count`.
`jsonl`	Newline-delimited JSON; one cell per line.
`junit`	JUnit XML for Jenkins / GitHub Actions test reporters.
`sarif`	SARIF 2.1.0 — feed into security dashboards.
`html`	Self-contained HTML report (one file, all assets inlined).
`csv`	One row per cell, header included.
`markdown`	Human-readable Markdown summary.
`github`	GitHub Actions workflow-command annotations (`::error::` / `::warning::`).

To render one run in several formats without re-running it, save it with --format json and use kanros report: kanros report --from run.json --format junit --output junit.xml.

Exit codes

Code	Meaning
`0`	All cells passed their assertions.
`1`	At least one cell failed or errored: a failing assertion, an assertion that could not run (for example a `custom`/`wasm` assertion without `--plugins`), a test with no assertions, or a provider error on that cell. For `redteam run`, a probe produced a finding.
`2`	Config or fatal error — the config could not be parsed or validated, a provider could not be constructed (for example an `env:NAME` credential whose variable is unset), a cache IO error, or a command-line usage error.

CI/CD pipelines can branch on these codes to distinguish "tests failed" from "your config is broken" from "the provider is down".

Tests without assertions

A test case with zero effective assertions — no assert: of its own and none inherited from default_test — checks nothing, so its cells are recorded as errors, never passes:

  [ERR ] echo:identity                   prompt#0  test#0 (unchecked)  say hi
         ! assert-set errored: this test has no assertions (no `assert:` on the test and none in `default_test`), so nothing checked the output; the cell is recorded as an error rather than a pass. Add at least one assertion.

The run exits 1. kanros validate warns about such tests up front:

warning: tests[0] ("unchecked") has no assertions (none of its own, none in `default_test`); `kanros run` records its cells as errors

Examples

Run the default config and emit a terminal table:

kanros run

Use a non-default config and write a JSONL stream:

kanros run --config configs/regression.yaml --format jsonl > results.jsonl

Disable cache and enable WASM plugins (typical CI invocation when you want to grade on a fresh response):

kanros run --no-cache --plugins

Use a CI-scoped cache database so the per-PR runner does not pollute the developer cache:

kanros run --cache-path ./.kanros-cache.db

What kanros run does

Load and validate the config. Config::load dispatches on file extension to YAML / TOML / JSON, then runs validate() (which enforces the "≥ 1 provider, ≥ 1 prompt, ≥ 1 test, family:model id shape" invariants).
Open the cache (unless --no-cache). The cache is keyed by (provider_id, request); a hit skips the provider call entirely. A miss runs the provider and stores the response.
Open the plugin host (if --plugins). WASM plugins are sandboxed by default — no network, no filesystem, no clock, no env.
Run the eval matrix. For each (prompt, test, provider) cell the runner renders the prompt, calls the provider with the configured retries/timeout, runs every assertion in tests[i].assert, and records the outcome.
Persist the run summary. The summary is stored in the cache database alongside cell-level results so kanros diff and kanros share can find it later. This happens even when --no-cache is set for the response cache.
Emit the report. The chosen format is written to stdout; logs go to stderr.

Performance knobs

The runner: block in kanros.yaml controls concurrency and timeouts:

runner:
  max_concurrency: 32          # global cap on in-flight provider calls
  rate_limit_per_provider: 0   # 0 = no per-provider RPS cap
  timeout_seconds: 120         # per-request timeout
  retries: 3                   # max retries on retryable errors

These are loaded into RunOptions and applied for the whole run. --concurrency N overrides max_concurrency for one invocation (also in --watch mode) — handy for debugging a provider with kanros run --concurrency 1.

Cancellation

kanros run runs the eval loop inside a tokio_util::sync::CancellationToken. SIGINT (Ctrl-C) cancels in-flight provider calls and writes a partial summary to stdout before exit. Cells already completed are reported as normal; cells in flight are reported as Error with a cancellation reason.

Caveats

The terminal format truncates long prompts and responses. Use json or jsonl if you need the full text.
The sarif format is intended for redteam run; passing it to kanros run will still produce valid SARIF but the result types are generic assertion failures rather than security findings.
--plugins enables the WASM plugin host. If the host fails to initialise (e.g. a wasmtime version mismatch), kanros warns and continues without plugins. type: custom and type: wasm assertions then return AssertionOutcome::Error.