Quickstart

This walks through the smallest possible kanros eval — one provider, one prompt, two test cases — and shows the expected output. It uses the built-in echo provider so no network and no API key are required.

The fastest path is the example bundled in the binary — two commands in an empty directory, nothing to download:

kanros init --example smoke && kanros run

kanros init --example smoke writes the config below to ./kanros.yaml (kanros init --list-examples lists every bundled example), and kanros run reads ./kanros.yaml by default. The rest of this page walks through that file.

1. Create a config

Run kanros init --example smoke, or save the following as kanros.yaml:

description: kanros end-to-end smoke test — no network, no credentials.

providers:
  - id: echo:identity

prompts:
  - "say {{ word }}"

tests:
  - description: case-insensitive substring match
    vars:
      word: hello
    assert:
      - type: icontains
        value: HELLO
      - type: length
        min: 1
        max: 100

  - description: structural checks
    vars:
      word: world
    assert:
      - type: starts-with
        value: "say "
      - type: contains
        value: world
      - type: word-count
        min: 2
        max: 2

The same file is bundled in the kanros binary (that is what kanros init --example smoke writes) and is exercised by the workspace's integration tests.

2. Run it

kanros run --config kanros.yaml

Expected output (timing varies):

kanros run 8c2c5c9e-…
  cells: 2 passed, 0 failed, 0 errored (3 ms)
  [PASS] echo:identity   prompt#0  test#0  say hello
  [PASS] echo:identity   prompt#0  test#1  say world

The echo:identity provider returns the rendered prompt verbatim. That is why icontains: HELLO passes against "say hello" (case-insensitive), and why the structural checks pass against "say world".

3. Try a different output format

Every report format is a flag on kanros run:

kanros run --format json     # pretty JSON for machine consumption
kanros run --format jsonl    # newline-delimited JSON, one cell per line
kanros run --format junit    # JUnit XML for Jenkins / GitHub test reporters
kanros run --format sarif    # SARIF 2.1.0
kanros run --format html     # self-contained HTML report
kanros run --format markdown # human-readable Markdown summary
kanros run --format csv      # CSV with header row
kanros run --format github   # GitHub Actions workflow annotations

CI pipelines typically capture two formats — one human-readable (markdown or terminal) for build logs and one machine-readable (junit, sarif, or jsonl) for downstream tooling.

4. Validate without running

To check that a config parses and validates without paying for any LLM calls:

kanros validate --config kanros.yaml
# ok: 1 providers, 1 prompts, 2 tests, 2 cells

Useful as a pre-commit hook on the YAML.

5. Inspect the JSON Schema

The kanros config has a published JSON Schema. Pipe it into your editor's schema-aware YAML support for live validation and autocomplete:

kanros schema > schemas/kanros.schema.json

Then point your editor at it — for VS Code, add the file to yaml.schemas in settings.json.

6. Switch to a real model

Swap the echo provider for a real one. Keep the same tests, pin the sampling parameters so reruns are comparable, and add a model-graded check:

providers:
  - id: openai:gpt-4o-mini
    temperature: 0
    seed: 42
    config:
      api_key: env:OPENAI_API_KEY   # read from the environment, never committed

prompts:
  - "say {{ word }}"

tests:
  - vars:
      word: hello
    assert:
      - type: icontains
        value: hello
      - type: llm-rubric
        value: The reply is a short, friendly greeting.
        provider: { id: "openai:gpt-4o-mini" }

export OPENAI_API_KEY=sk-...        # or: kanros auth set openai
kanros run

No API key? Run a local model with Ollama instead: ollama pull llama3.1, then use id: ollama:llama3.1 (no key needed).

Every run is recorded, so you can compare the last two runs cell by cell:

kanros diff latest~1 latest

Next steps

Troubleshooting

  • error: cannot read config file 'kanros.yaml' — kanros could not open the file. Check the --config path; the default is ./kanros.yaml.
  • error: invalid YAML in kanros.yaml — the YAML parser rejected the file. The error message includes the line/column from serde_yaml.
  • Provider error — if you wired up a network provider (openai, anthropic, …) without credentials, the run stops before any cell with exit code 2; errors returned by the provider during a cell mark that cell errored and the run exits with 1. See the provider's section in Providers for expected environment variables and credential lookup order.