CI integration

Run your eval suite on every pull request and fail the build when it regresses. kanros is one static binary with CI-friendly exit codes, so any CI system works. This page covers GitHub Actions, GitLab CI and a generic recipe.

Exit codes (kanros run, kanros report):

GitHub Actions

The kanros action runs the eval once, writes a JSON summary, annotates failing cells inline on the pull request and appends a results table to the job summary.

name: evals
on: [pull_request]

jobs:
  kanros:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - id: kanros
        uses: kanros/kanros-action@v1
        with:
          config: evals/kanros.yaml
          format: github        # inline annotations; or markdown, junit, sarif
          max-cost: "2.00"      # stop spending after $2
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - if: always()
        run: echo "passed=${{ steps.kanros.outputs.pass-count }} failed=${{ steps.kanros.outputs.fail-count }}"

• Pin the action to a release tag: each tag pulls the matching ghcr.io/kanros/kanros:v<version> image.
• Any provider key can be passed through env: on the step.
• Set fail-on-error: "false" for advisory runs that report but never fail the job.
• Outputs: pass-count, fail-count, error-count, run-id, summary-json-path. The full input reference is in the action README.

Without the action

If you prefer plain steps, for example to upload JUnit results to a test reporter:

      - name: Install kanros
        run: curl -fsSL https://kanros.dev/install.sh | KANROS_VERSION=v0.1.0-rc.4 sh
      - name: Run evals
        run: |
          ~/.kanros/bin/kanros run --config evals/kanros.yaml --format json > run.json \
            || echo "EVAL_FAILED=1" >> "$GITHUB_ENV"
          ~/.kanros/bin/kanros report --from run.json --format junit --output junit.xml --exit-zero
          ~/.kanros/bin/kanros report --from run.json --format github --exit-zero
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: kanros-junit
          path: junit.xml
      - name: Fail on eval regressions
        if: env.EVAL_FAILED == '1'
        run: exit 1

The eval runs once. kanros report --from renders the saved result in other formats without calling any provider.

GitLab CI

evals:
  image: debian:bookworm-slim
  before_script:
    - apt-get update -qq && apt-get install -y -qq curl xz-utils ca-certificates
    - curl -fsSL https://kanros.dev/install.sh | KANROS_VERSION=v0.1.0-rc.4 sh
    - export PATH="$HOME/.kanros/bin:$PATH"
  script:
    - kanros run --config evals/kanros.yaml --format json > run.json || status=$?
    - kanros report --from run.json --format junit --output junit.xml --exit-zero
    - exit ${status:-0}
  artifacts:
    when: always
    reports:
      junit: junit.xml

Store provider keys as masked CI/CD variables (for example OPENAI_API_KEY). GitLab shows each failing cell in the merge request's test report.

Any other CI system

1. Install a pinned version: curl -fsSL https://kanros.dev/install.sh | KANROS_VERSION=v0.1.0-rc.4 sh. The installer verifies the SHA-256 checksum. On Windows use irm https://kanros.dev/install.ps1 | iex.
2. Run once to JSON: kanros run --format json > run.json, and keep the exit code.
3. Render whatever your CI understands: kanros report --from run.json --format junit|sarif|markdown|html.
4. Exit with the code from step 2.

Keeping CI fast and cheap

• Cache responses. Unchanged prompt, provider and parameter combinations are served from the local response cache, keyed on everything that affects the request. Persist the cache directory between CI runs (--cache-path) to skip them. Use --no-cache for runs that must hit the provider.
• Pin sampling. Set temperature: 0 and, where the provider supports it, seed on each provider so reruns are comparable.
• Cap spend with --max-cost <USD> (or the action's max-cost input).
• Compare against the last run. kanros diff latest~1 latest lists the cells that changed between two recorded runs.