Batch API

For long-running evaluations against OpenAI you can ship the entire eval matrix as a single batch job. Batch jobs are cheaper (50 % off list price) and have a 24-hour SLA, so they're ideal for nightly regression suites that don't need real-time feedback.

In-process batch (kanros run --batch)

$ kanros run --batch --batch-poll-interval 30
Submitted batch: batch_abc123
status=in_progress progress=12/500
status=in_progress progress=412/500
status=completed progress=500/500

This:

  1. Serialises every (prompt × test × provider) cell into a single JSONL file (one chat-completion request per line).
  2. Uploads the file to POST /v1/files and creates a batch job via POST /v1/batches with completion_window: "24h".
  3. Polls GET /v1/batches/{id} every --batch-poll-interval seconds (default 30) until the job is completed, failed, expired, or cancelled.
  4. Downloads the per-line results and grades each response with the test's assertions exactly like kanros run does — same pass / fail / error rules (a test with no assertions is an error), same cost and token accounting — then emits the run summary as if it had run synchronously. A non-2xx or error line, or a cell the batch never answered, is an errored cell.

The exit code matches kanros run: 0 only when every cell passed, 1 when any cell failed or errored.

Mixed-provider configs are rejected at submit time — the batch API ships one provider family per job. v1 supports openai only.

Fire-and-forget (kanros batch)

For workflows where the submitting machine doesn't stay online, the kanros batch subcommands separate submission from result fetching:

# 1. Submit and capture the id.
$ ID=$(kanros batch submit --config kanros.yaml)

# 2. Later (from any machine with the same config), check progress.
$ kanros batch status "$ID"
status=in_progress progress=412/500

# 3. When complete, download and grade the results.
$ kanros batch fetch "$ID" --config kanros.yaml
kanros run 0b6c…
  cells: 498 passed, 2 failed, 0 errored (0 ms)
  …

fetch grades against the config it is given, so use the same config (and the same tests) you submitted; it exits 0 / 1 like kanros run.

Each submit line in the input file is correlated with its output by a custom_id of the shape p{prompt_index}-t{test_index}-{provider_id}, so the cell origin survives the round trip.

Implementation notes

  • Multipart upload is hand-rolled — the kanros workspace does not opt into reqwest's multipart feature.
  • Polling uses tokio's sleep and is cancellation-safe.
  • The first failure to fetch the result file aborts the run; partial results are not currently merged into the cache.