v0.1.0-rc.1 — free during early access
Evaluate & red-team LLM apps at Rust speed
A fast, type-safe framework for testing LLM applications. Single binary. Local-first. No telemetry, no cloud account — your prompts, datasets, and results never leave your machine.
Why kanros
Built like infrastructure,
not a script.
Every provider, assertion, and report formatter compiles against a single frozen trait layer. No string-typed plugin bus. No runtime schema drift.
Rust speed
Eval cells run in a bounded tokio pool with per-provider rate limits, retries, native SSE streaming, and Ctrl+C cancellation. A SQLite response cache makes repeat runs nearly free.
Type-safe by construction
The frozen kanros-core traits — Provider, Assertion, Config, EvalResult — are the single source of truth the whole workspace compiles against.
Local-first & private
One static executable. Configs, prompts, datasets, and the cache live on your filesystem. No telemetry, no SaaS account — ever.
Sandboxed plugins
Custom graders run in a WASM Component Model host (wasmtime) or an embedded JS engine (boa) — both default-deny on filesystem, network, clock, and env.
TUI + Web UI
Watch runs live in a ratatui terminal view, or use the embedded Web UI with SSE live tail, run history, and side-by-side run diffs. Optionally share results via a self-hostable share endpoint.
CI-native reports
Exit codes for pipelines; JUnit for test dashboards; SARIF 2.1.0 for code scanning; plus JSON, JSONL, HTML, Markdown, CSV, and GitHub Actions annotations.
Red-team fleet
Attack your app before someone else does.
16 adversarial plugins wired through 6 attack strategies probe your application for the failure modes that matter — locally, against the same config that runs your evals. Probes are deterministic given a seed, so findings reproduce.
Plugins
Strategies
Configuration
YAML in.
Everything out.
One declarative file describes providers, prompts, tests, and assertions. TOML and JSON work too — all three parse through the same serde struct.
- 44 assertions — deterministic, similarity, model-graded, BLEU/ROUGE/METEOR, code, composite, safety, and Ragas-style RAG checks.
- minijinja templating with URI loaders, plus CSV, JSONL, and HuggingFace dataset loaders.
- Validate before you spend —
kanros validateparses and checks the config without calling a single provider. - Bring existing configs —
kanros migrateconverts declarative eval configs from other tools.
providers: - id: openai:gpt-5.2 - id: anthropic:claude-fable-5 prompts: - "summarize: {{ article }}" tests: - vars: article: file://data/report.txt assert: - type: contains value: "key findings" - type: llm-rubric value: "faithful to the source" - type: length max: 1000
27 providers
Every model you run. One interface.
Cloud APIs, local backends, and generic custom protocols — all behind the same frozen Provider trait.
…plus the built-in echo provider, so the smoke test runs with no API key and no network.
Reports
Speaks your pipeline's language.
Nine output formats, from a human-readable terminal summary to SARIF 2.1.0 for security dashboards.
Get started
From zero to first eval
in under a minute.
Pre-built binaries for macOS (arm64 + x86_64), Linux (glibc + musl, x86_64 + arm64), and Windows — with SHA256 checksums on every release. Free during early access.
$ curl --proto '=https' --tlsv1.2 -LsSf https://kanros.dev/install.sh | sh $ kanros run -c examples/smoke/kanros.yaml
PS> irm https://kanros.dev/install.ps1 | iex PS> kanros run -c examples/smoke/kanros.yaml
# Grab the archive for your platform from kanros.dev/download $ sha256sum -c kanros-<target>.tar.xz.sha256 $ tar -xJf kanros-<target>.tar.xz
The smoke fixture uses the built-in echo provider — full pipeline, no API key, no network round-trip. All builds and checksums are on the download page.