Configuration

kanros configuration is one declarative file in YAML (default), TOML, or JSON. All three formats deserialize into the same Config struct via serde and are accepted interchangeably by every subcommand that takes --config.

The schema is derived directly from the Rust types in the frozen kanros-core trait layer. Run kanros schema to print the live JSON Schema.

Top-level shape

description: optional free-form description
providers:        # required, ≥ 1
  - id: openai:gpt-4o-mini
prompts:          # required, ≥ 1
  - "Summarize: {{ article }}"
tests:            # required, ≥ 1
  - vars: { article: "…" }
    assert:
      - type: contains
        value: "key findings"
default_test:     # optional
  vars: {}
  assert: []
red_team:         # optional
  plugins: ["prompt-injection", "harmful-content"]
  strategies: ["base64"]
  num_tests: 5
runner:           # optional
  max_concurrency: 32
  rate_limit_per_provider: 0
  timeout_seconds: 120
  retries: 3

The config is validated semantically after deserialisation. The three hard requirements are: at least one provider, at least one prompt, and at least one test. Every provider id must contain a colon (the segment before the colon is the provider family).

providers

Each entry is a ProviderRef:

providers:
  - id: openai:gpt-4o-mini
    config:
      temperature: 0.0
      max_tokens: 512

• id: the provider identifier in family:model form. The family (openai, anthropic, ollama, echo, …) selects which provider crate to load; the model is forwarded to the provider verbatim.
• config: an arbitrary JSON object passed through to the provider unchanged. Each provider documents its own config keys.

prompts

A list of prompt specs. Each entry is either a literal string with minijinja-style template variables, or a URI:

prompts:
  - "Summarize the article: {{ article }}"
  - "file://prompts/system.txt"
  - "https://example.com/templates/v2.j2"
  - "package://kanros-prompt-pack-foo@1.2.3"

URI schemes recognised today: file://, http://, https://, package://.

tests

The test matrix. Each TestCase carries:

tests:
  - description: optional human label
    vars:
      article: "{{ file://articles/launch.md }}"
      audience: "developers"
    assert:
      - type: contains
        value: "key findings"
      - type: latency
        max_ms: 5000
    tags:
      - smoke
      - regression

• vars: a JSON object of template variables for prompt rendering.
• assert: a list of assertions (see Assertions).
• tags: arbitrary string tags, filterable from the CLI.

The total number of cells executed is prompts × tests × providers. A 4-prompt, 8-test, 3-provider config runs 96 cells.

default_test

Defaults merged into every test case. Per-test fields take precedence; default_test.assert is appended to each test's assert list (not replaced).

default_test:
  vars:
    audience: "developers"
  assert:
    - type: latency
      max_ms: 10000   # global SLA every test inherits

red_team

Optional red-team configuration consumed by kanros redteam run / kanros redteam generate. See Red Teaming.

red_team:
  plugins:
    - prompt-injection
    - harmful-content
  strategies:
    - base64
    - leetspeak
  num_tests: 10

runner

Per-run knobs for the eval loop:

Multiple formats

The same fields are accepted in TOML or JSON:

description = "regression"

[[providers]]
id = "openai:gpt-4o-mini"

[[prompts]]
value = "Summarize: {{ article }}"

[[tests]]
description = "case 1"
vars = { article = "…" }

[[tests.assert]]
type = "contains"
value = "key findings"

{
  "providers": [{ "id": "openai:gpt-4o-mini" }],
  "prompts": ["Summarize: {{ article }}"],
  "tests": [
    {
      "vars": { "article": "…" },
      "assert": [{ "type": "contains", "value": "key findings" }]
    }
  ]
}

Strict parsing

The root Config struct is #[serde(deny_unknown_fields)], so a typo in a top-level key surfaces as a clean parse error rather than being silently ignored. Inner structs follow the same convention where it makes sense.

Error Classes

kanros is contractually required (P0 §11 verification bar) to surface at least ten distinct, named parse-time configuration error classes through the public kanros-core API. The full battery lives in crates/kanros-error-fixtures/tests/error_class_coverage.rs; the test test_at_least_ten_error_classes runs each fixture, classifies the returned ConfigError, and prints a coverage table.

The six ConfigError variants are the top-level classification — many user-visible parse errors share the Yaml (or Toml / Json) variant and are further distinguished by stable substrings of the wrapped serde diagnostic. The classes below are the ones the §11 test verifies as reachable today:

The six top-level ConfigError variants are:

• ConfigError::Yaml — YAML parse failure, wraps serde_yaml::Error.
• ConfigError::Toml — TOML parse failure, wraps toml::de::Error.
• ConfigError::Json — JSON parse failure, wraps serde_json::Error.
• ConfigError::Io — file read failure, wraps std::io::Error.
• ConfigError::Validation — semantic checks after a successful parse.
• ConfigError::UnknownFormat — file extension not recognised.

All variants implement miette::Diagnostic and carry a stable kanros::config::* error code (e.g. kanros::config::yaml, kanros::config::validation) so downstream tooling can dispatch on the code rather than scraping to_string().

See also

• Providers — the list of provider families bundled with kanros.
• Assertions — the full assertion catalogue.
• kanros run — runtime flags that override config defaults.