Subprocess provider

The subprocess provider runs your own program as the model under test: a Python script wrapping an agent, a Node service client, a shell pipeline — anything that can read stdin and write stdout. kanros starts the program once per request, writes one JSON request to its stdin, and reads one JSON reply from its stdout.

providers:
  - id: subprocess:my-bot              # `subprocess:<any-name>`
    config:
      command: ["python3", "bots/my_bot.py"]   # required
      env:                                       # optional
        BOT_MODE: eval
      cwd: .                                     # optional
      timeout_ms: 60000                          # optional (default 60000)

Config

Key	Type	Default	Meaning
`command`	list of strings	required	argv of the program. The first element is the executable (looked up on `PATH`), the rest are its arguments. It is not run through a shell: no globbing, pipes or `$VARS` — use `["sh", "-c", "…"]` if you need one. Must be non-empty.
`env`	map string→string	`{}`	Extra environment variables, merged on top of kanros' own environment (the program inherits everything else, including API keys).
`cwd`	string	kanros' working directory	Working directory of the program. Relative paths in `command` resolve against it — not against the config file's directory.
`timeout_ms`	integer	`60000`	Per-request deadline for the program to finish writing stdout.

Unknown keys are rejected. The generation parameters set on the provider entry (temperature, max_tokens, top_p, stop, seed, response_format) are forwarded to your program in the request (below).

Input contract (kanros → your program)

For every cell (and every retry), kanros spawns command with no extra arguments and no extra environment variables, writes one line of JSON to stdin, then closes stdin (EOF):

{
  "messages": [
    {"role": "user", "content": "say hello", "tool_calls": [], "name": null}
  ],
  "max_tokens": null,
  "temperature": 0.0,
  "top_p": null,
  "stop": [],
  "seed": 7,
  "tools": [],
  "json_mode": false,
  "extra": null
}

Field	Type	Notes
`messages`	list of messages	The rendered prompt. A text prompt is one `user` message; a chat prompt keeps its `system` / `user` / `assistant` turns in order. Each message has `role`, `content` (string), `tool_calls` (list) and `name` (string or `null`). The prompt to answer is the last `user` message.
`max_tokens`	integer or `null`	From the provider entry, else `null`.
`temperature`	number or `null`	Same.
`top_p`	number or `null`	Same.
`stop`	list of strings	Stop sequences (empty when unset).
`seed`	integer	Omitted when unset.
`tools`	list of `{name, description, parameters}`	Tool specs, if any.
`json_mode`	boolean	`true` when the provider entry requests JSON output (`response_format: {type: json_object}` or a schema).
`extra`	any JSON	Provider-specific extras (normally `null`).

Read stdin to EOF (or read one line) and ignore fields you do not need; more fields may be added in future releases.

Output contract (your program → kanros)

Print exactly one JSON object to stdout, then exit with status 0. The reply text goes in content:

{"content": "hello!"}

That is the whole required contract. Optional fields:

Field	Type	Default	Notes
`content`	string	required	The model's reply; what assertions check. `""` is allowed and means an intentionally empty reply. Required unless `tool_calls` is non-empty.
`tool_calls`	list of `{id, name, arguments}`	`[]`	For tool-calling turns. `arguments` is any JSON.
`finish_reason`	`"stop"` \| `"length"` \| `"tool_call"` \| `"content_filter"` \| `"other"`	`"stop"`
`usage`	`{"prompt_tokens": n, "completion_tokens": n, "total_tokens": n}`	none	Token counts; when present all three integers are required. Shown in reports and summed into the run's token totals.
`model`	string	the provider id	Model name recorded in results.
`latency_ms`	integer	measured wall-clock time	Override when your program knows the true model latency.
`raw`	any JSON	`null`	Stored verbatim with the result for debugging.

Unknown fields are ignored. There is no plain-text mode: stdout must be JSON. Write logs and debug output to stderr — kanros passes your program's stderr through to the terminal.

Cost. There is no cost field. kanros prices calls from usage with a built-in price table keyed by provider id, and subprocess:* ids are not priced, so cost is unknown for these cells: a cost assertion reports skipped, and a cell whose only assertion was skipped is an error. Use latency or deterministic assertions instead.

What counts as an error

Each of these fails the cell (status ERR, run exits 1) with a message naming the problem — none of them is ever treated as an empty reply:

Situation	Error
No `content` key (e.g. you printed `{"output": "…"}`), or `content: null` without `tool_calls`	`subprocess reply has no` content`string (expected {"content": "<reply text>"}, or a non-empty`tool_calls`list); keys present:`output``
`content` is not a string	`subprocess reply key` content `must be a string, got a number`
stdout is empty	`subprocess wrote nothing to stdout; …`
stdout is not JSON (e.g. a stray `print()` before the reply)	`subprocess stdout was not valid JSON (…); … print logs to stderr, not stdout. stdout began: "…"`
stdout is JSON but not an object	`subprocess stdout must be a JSON object …, got an array`
An optional field has the wrong shape (e.g. `"usage": 5`)	`subprocess reply has a malformed field: …`
Non-zero exit status (or killed by a signal)	`subprocess exited with non-zero status N; check stderr for details`
The executable cannot be started	`failed to spawn` …`: …` (a network-class error, retried)
`timeout_ms` elapses before stdout closes	the process is killed (`SIGKILL` / `TerminateProcess`) and the call times out (retried per `runner.retries`)

The runner's own per-call timeout (runner.timeout_seconds) also applies, whichever is shorter.

A working example

bots/echo_bot.py — answers with the last user message and reports token usage:

#!/usr/bin/env python3
import json
import sys

req = json.loads(sys.stdin.readline())
prompt = next(m["content"] for m in reversed(req["messages"]) if m["role"] == "user")

print(f"debug: got {len(req['messages'])} message(s)", file=sys.stderr)  # logs → stderr

reply = f"you said: {prompt}"
print(json.dumps({
    "content": reply,
    "model": "echo-bot-v1",
    "usage": {
        "prompt_tokens": len(prompt.split()),
        "completion_tokens": len(reply.split()),
        "total_tokens": len(prompt.split()) + len(reply.split()),
    },
}))

kanros.yaml:

providers:
  - id: subprocess:echo-bot
    config:
      command: ["python3", "bots/echo_bot.py"]
      timeout_ms: 10000

prompts:
  - "Tell me about {{ topic }}"

tests:
  - vars: {topic: large language models}
    assert:
      - type: starts-with
        value: "you said: "
      - type: contains
        value: large language models

Run it from the directory that contains bots/:

kanros run --no-cache

While developing the script, pass --no-cache: responses are cached by provider id, config and request, so an unchanged prompt would otherwise be answered from the cache without running your updated program.

Notes

One process per request: with runner.max_concurrency: N, up to N copies of your program run at once. Keep start-up cheap, or have the script forward to a long-running local server.
The provider does not advertise streaming, tool use, JSON mode or vision capabilities; your program receives tools / json_mode and may honour them, but kanros does not require it.
Migrating from promptfoo: kanros migrate turns exec: and file://….py providers into subprocess entries with the same command, plus a # TODO(migrate): reminder — promptfoo passes the prompt as an argument and reads plain-text stdout, so the script needs adapting to the JSON contract above.