Introduction

kanros is a fast, type-safe Rust framework for evaluating and red-teaming LLM applications. It ships as a single static binary, runs locally, sends no telemetry, and requires no cloud account.

The workflow is declarative: a YAML config describes providers, prompts, and tests; the kanros binary runs the matrix from the command line and emits machine-readable reports that CI pipelines consume directly.

What kanros does

You describe an evaluation suite in YAML (or TOML/JSON):

providers:
  - id: openai:gpt-4o-mini
  - id: anthropic:claude-3-5-haiku
prompts:
  - "Summarize the following article: {{ article }}"
tests:
  - vars:
      article: "{{ file://articles/launch.md }}"
    assert:
      - type: contains
        value: "key findings"
      - type: llm-rubric
        value: "the summary is faithful and does not invent facts"
        provider: { id: openai:gpt-4o }

Then you run it:

kanros run --config kanros.yaml

kanros expands the prompts × providers × tests matrix into cells, executes each cell with bounded concurrency, runs every assertion, optionally consults a SQLite-backed cache, and emits a structured report (terminal, JSON, JSONL, JUnit XML, SARIF 2.1.0, HTML, CSV, Markdown, or GitHub Actions annotations).

Why kanros is different

  • Single static binary, no runtime. No Node.js, no Python, no Docker. Run the one-line installer or download a prebuilt build and you are done. The same binary runs on macOS Intel, macOS Apple Silicon, Linux glibc/musl, Windows MSVC, and Linux ARM64.

  • Type-safe core with a frozen trait layer. The kanros-core crate pins the contract between the runner, providers, and assertions behind a semver-stable trait surface. Provider crates and assertion families compose against it without monkey-patching, runtime reflection, or duck typing.

  • CI-friendly outputs and exit codes. Every report format is machine-readable. Exit codes follow the spec: 0 for all-pass, 1 for assertion failures, 2 for config errors, 3 for provider/runtime errors. Drop kanros into a pipeline without writing glue code.

When to use kanros

  • You ship an LLM-powered feature and want regression tests that fail a PR when output quality drops.
  • You run red-team probes as part of your release gating and need SARIF output to feed into a security dashboard.
  • You evaluate RAG pipelines and want context-recall / context-precision / faithfulness metrics that work the same way locally and in CI.
  • You compare multiple providers on the same prompts to inform pricing or latency decisions.
  • You score agent traces (multi-step tool-call transcripts) and need a consistent way to grade tool-use behaviour.

Status and licensing

kanros is at version v0.1.0-rc.1, a release candidate in its soak period before promotion to v0.1.0 GA. The trait layer in kanros-core is frozen behind a semver-stability gate; provider and assertion implementations are evolving inside their respective crates.

kanros is proprietary software and is free to use during early access, for both personal and commercial work — no license key, no account, no telemetry. See Installation for details.