Quickstart

This walks through the smallest possible kanros eval — one provider, one prompt, two test cases — and shows the expected output. It uses the built-in echo provider so no network and no API key are required.

1. Create a config

Save the following as kanros.yaml:

description: kanros end-to-end smoke test — no network, no credentials.

providers:
  - id: echo:identity

prompts:
  - "say {{ word }}"

tests:
  - description: case-insensitive substring match
    vars:
      word: hello
    assert:
      - type: icontains
        value: HELLO
      - type: length
        min: 1
        max: 100

  - description: structural checks
    vars:
      word: world
    assert:
      - type: starts-with
        value: "say "
      - type: contains
        value: world
      - type: word-count
        min: 2
        max: 2

The same file ships at examples/smoke/kanros.yaml in the repo and is exercised by the workspace's integration tests.

2. Run it

kanros run --config kanros.yaml

Expected output (timing varies):

kanros run 8c2c5c9e-…
  cells: 2 passed, 0 failed, 0 errored (3 ms)
  [PASS] echo:identity   prompt#0  test#0  say hello
  [PASS] echo:identity   prompt#0  test#1  say world

The echo:identity provider returns the rendered prompt verbatim. That is why icontains: HELLO passes against "say hello" (case-insensitive), and why the structural checks pass against "say world".

3. Try a different output format

Every report format is a flag on kanros run:

kanros run --format json     # pretty JSON for machine consumption
kanros run --format jsonl    # newline-delimited JSON, one cell per line
kanros run --format junit    # JUnit XML for Jenkins / GitHub test reporters
kanros run --format sarif    # SARIF 2.1.0
kanros run --format html     # self-contained HTML report
kanros run --format markdown # human-readable Markdown summary
kanros run --format csv      # CSV with header row
kanros run --format github   # GitHub Actions workflow annotations

CI pipelines typically capture two formats — one human-readable (markdown or terminal) for build logs and one machine-readable (junit, sarif, or jsonl) for downstream tooling.

4. Validate without running

To check that a config parses and validates without paying for any LLM calls:

kanros validate --config kanros.yaml
# ok: 1 providers, 1 prompts, 2 tests, 2 cells

Useful as a pre-commit hook on the YAML.

5. Inspect the JSON Schema

The kanros config has a published JSON Schema. Pipe it into your editor's schema-aware YAML support for live validation and autocomplete:

kanros schema > schemas/kanros.schema.json

Then point your editor at it — for VS Code, add the file to yaml.schemas in settings.json.

Next steps

• Add a real provider in Providers.
• Browse the full assertion catalogue in Assertions.
• Already have eval configs from another tool? Convert them with kanros migrate.
• Wire a red-team probe via Red Teaming.

Troubleshooting

• error: cannot read config file 'kanros.yaml' — kanros could not open the file. Check the --config path; the default is ./kanros.yaml.
• error: invalid YAML in kanros.yaml — the YAML parser rejected the file. The error message includes the line/column from serde_yaml.
• Provider error — if you wired up a network provider (openai, anthropic, …) and see a provider error, the exit code will be 3. See the provider's section in Providers for expected environment variables and credential lookup order.