Configuration

kanros configuration is one declarative file in YAML (default), TOML, or JSON. All three formats deserialize into the same Config struct via serde and are accepted interchangeably by every subcommand that takes --config.

The schema is derived directly from the Rust types in the frozen kanros-core trait layer. Run kanros schema to print the live JSON Schema.

Top-level shape

description: optional free-form description
providers:        # required, ≥ 1
  - id: openai:gpt-4o-mini
prompts:          # required, ≥ 1
  - "Summarize: {{ article }}"
tests:            # required, ≥ 1 (inline cases and/or file:// URIs)
  - vars: { article: "…" }
    assert:
      - type: contains
        value: "key findings"
default_test:     # optional
  vars: {}
  assert: []
red_team:         # optional
  plugins: ["prompt-injection", "harmful-content"]
  strategies: ["base64"]
  num_tests: 5
runner:           # optional
  max_concurrency: 32
  rate_limit_per_provider: 0
  timeout_seconds: 120
  retries: 3

The config is validated semantically after deserialisation. The three hard requirements are: at least one provider, at least one prompt, and at least one test. Every provider id must contain a colon (the segment before the colon is the provider family).

providers

Each entry is a ProviderRef:

providers:
  - id: openai:gpt-4o-mini
    temperature: 0
    max_tokens: 512
    config:
      api_key: env:OPENAI_API_KEY
      base_url: "https://api.openai.com/v1"
      timeout_ms: 30000

• id: the provider identifier in family:model form. The family (openai, anthropic, ollama, echo, …) selects which provider crate to load; the model is forwarded to the provider verbatim.
• config: a JSON object passed through to the provider. Each provider documents its own config keys, and each rejects keys it does not recognise — a typo fails the run instead of being silently dropped. Keys are not shared across families, with one exception: the generation keys below are accepted for every family, either inside config: or at provider level. Credential keys (api_key, api_token, access_key_id, secret_access_key, session_token) accept env:NAME to read the value from an environment variable; an unset variable is a hard error.
• Generation keys — temperature (>= 0), top_p (0–1), max_tokens (integer >= 1), seed (integer >= 0), stop (string or list), response_schema (JSON Schema object) and response_format ({type: text | json_object | json_schema}). A provider-level key overrides the same key in config:. Invalid values fail kanros validate. See Providers.
• fallback: optional list of provider ids to try, in order, when this provider fails with a retryable error after its retries are exhausted.

prompts

A list of prompt specs. Each entry is either a literal string with minijinja-style template variables, or a URI:

prompts:
  - "Summarize the article: {{ article }}"
  - "file://prompts/system.txt"
  - "https://example.com/templates/v2.j2"
  - "package://kanros-prompt-pack-foo@1.2.3"

URI schemes recognised today: file://, http://, https://, package://.

Prompts are rendered with minijinja (Jinja2 syntax) against the test's vars: {{q}} and {{ q }} are equivalent, dotted paths ({{ user.name }}), filters ({{ q | upper }}) and blocks ({% if context %}…{% endif %}) work. Printing a variable the test does not define is an error — the cell is reported as error and the provider is never called — rather than sending a literal {{ q }} to the model. Testing an optional variable with {% if %} is fine. To send literal braces, wrap them in {% raw %}…{% endraw %}.

Assertion values (value, values, pattern, reference, context, rubric) are rendered against the same vars, so { type: contains, value: "{{ expected }}" } checks the test's expected var. Code (javascript, python) and URLs are not templated.

A prompt can attach images with the {{ image:PATH }} marker (PNG, JPEG, GIF or WebP, resolved relative to the config file) — see Vision input.

tests

The test matrix. Each TestCase carries:

tests:
  - description: optional human label
    vars:
      article: "{{ file://articles/launch.md }}"
      audience: "developers"
    assert:
      - type: contains
        value: "key findings"
      - type: latency
        max_ms: 5000
    tags:
      - smoke
      - regression

• vars: a JSON object of template variables for prompt rendering.
• assert: a list of assertions (see Assertions).
• tags: arbitrary string tags, filterable from the CLI.

The total number of cells executed is prompts × tests × providers. A 4-prompt, 8-test, 3-provider config runs 96 cells.

Loading tests from files

tests: can also point at files or datasets instead of (or alongside) inline cases. Use a single URI:

tests: file://cases.csv

or mix URI strings with inline cases in the list — loaded cases are spliced in at the position of their URI:

tests:
  - file://data/smoke.csv
  - description: hand-written edge case
    vars: { question: "" }
  - file://data/regressions.jsonl
  - huggingface://owner/name?split=test&subset=default

Relative file:// paths resolve against the directory holding the config file, not the shell's working directory. default_test vars and assertions apply to loaded cases exactly as they do to inline ones.

Files are loaded when the config is loaded — by kanros run (including --watch and --batch), kanros validate, and kanros redteam run. kanros validate reports how many cases came from files:

ok: 1 providers, 1 prompts, 6 tests (5 loaded from 3 test files), 6 cells

A missing file or a parse error fails with exit code 2 and names both the URI and the resolved path, e.g. cannot load tests from 'file://bad.jsonl' (/abs/path/bad.jsonl): format error: line 2: invalid JSON …. A string entry that is not a file:// or huggingface:// URI is a parse error. A runnable example lives in examples/tests-from-files/.

default_test

Defaults merged into every test case. default_test.vars fill in any variable a test does not set (per-test values take precedence), and default_test.assert is appended to each test's assert list (not replaced).

default_test:
  vars:
    audience: "developers"
  assert:
    - type: latency
      max_ms: 10000   # global SLA every test inherits

red_team

Optional red-team configuration consumed by kanros redteam run / kanros redteam generate. See Red Teaming.

red_team:
  plugins:
    - prompt-injection
    - harmful-content
  strategies:
    - base64
    - leetspeak
  num_tests: 10
  grader:                        # optional; default mode: heuristic
    mode: hybrid                 # heuristic | model | hybrid
    provider: openai:gpt-4o-mini # required for model / hybrid

runner

Per-run knobs for the eval loop:

kanros run --concurrency N overrides runner.max_concurrency for a single run (including --watch mode), and kanros run --max-cost USD overrides runner.max_cost_usd.

Multiple formats

The same fields are accepted in TOML or JSON:

description = "regression"
prompts = ["Summarize: {{ article }}"]

[[providers]]
id = "openai:gpt-4o-mini"

[[tests]]
description = "case 1"
vars = { article = "…" }

[[tests.assert]]
type = "contains"
value = "key findings"

{
  "providers": [{ "id": "openai:gpt-4o-mini" }],
  "prompts": ["Summarize: {{ article }}"],
  "tests": [
    {
      "vars": { "article": "…" },
      "assert": [{ "type": "contains", "value": "key findings" }]
    }
  ]
}

Strict parsing

The root Config struct is #[serde(deny_unknown_fields)], so a typo in a top-level key surfaces as a clean parse error rather than being silently ignored. Inner structs follow the same convention where it makes sense.

Error Classes

kanros is contractually required (P0 §11 verification bar) to surface at least ten distinct, named parse-time configuration error classes through the public kanros-core API. The full battery lives in crates/kanros-error-fixtures/tests/error_class_coverage.rs; the test test_at_least_ten_error_classes runs each fixture, classifies the returned ConfigError, and prints a coverage table.

The six ConfigError variants are the top-level classification — many user-visible parse errors share the Yaml (or Toml / Json) variant and are further distinguished by stable substrings of the wrapped serde diagnostic. The classes below are the ones the §11 test verifies as reachable today:

The six top-level ConfigError variants are:

• ConfigError::Yaml — YAML parse failure, wraps serde_yaml::Error.
• ConfigError::Toml — TOML parse failure, wraps toml::de::Error.
• ConfigError::Json — JSON parse failure, wraps serde_json::Error.
• ConfigError::Io — file read failure, wraps std::io::Error.
• ConfigError::Validation — semantic checks after a successful parse.
• ConfigError::UnknownFormat — file extension not recognised.

All variants implement miette::Diagnostic and carry a stable kanros::config::* error code (e.g. kanros::config::yaml, kanros::config::validation) so downstream tooling can dispatch on the code rather than scraping to_string().

See also

• Providers — the list of provider families bundled with kanros.
• Assertions — the full assertion catalogue.
• kanros run — runtime flags that override config defaults.