Agent traces

When a model uses tools — calling a function, hitting an API, looking something up in a vector store — a single output string is no longer enough to grade behaviour. You need to look at the trace: the sequence of turns the agent took, the tool calls it issued, and how it recovered from errors.

The kanros-agent crate provides the canonical in-memory shape of an agent trace plus a handful of small metric functions that scoring pipelines compose into assertions.

Anatomy of a trace

An AgentTrace is a vector of AgentTurns. Each turn carries a role (user, assistant, tool, system) and, for assistant turns, an optional list of ToolCalls.

use kanros_agent::{AgentTrace, AgentTurn, TurnRole, ToolCall};
use serde_json::json;

let trace = AgentTrace {
    turns: vec![
        AgentTurn { role: TurnRole::User, content: "What's the weather in Tokyo?".into(), tool_calls: vec![] },
        AgentTurn {
            role: TurnRole::Assistant,
            content: "".into(),
            tool_calls: vec![ToolCall {
                name: "get_weather".into(),
                arguments: json!({"city": "Tokyo"}),
                result: Some("18C, cloudy".into()),
                error: None,
            }],
        },
        AgentTurn { role: TurnRole::Assistant, content: "It's 18C and cloudy in Tokyo.".into(), tool_calls: vec![] },
    ],
};

A ToolCall carries:

name — the tool/function the model asked to invoke.
arguments — the raw JSON arguments (preserved so we do not lose number precision or object key ordering).
result — stringified successful result, or None.
error — error message if the tool errored. Both result and error being None represents a mid-stream snapshot where the call has been requested but not yet resolved.

Parsing OpenAI-style traces

If you already capture chat transcripts in OpenAI messages shape, parse_openai_style_trace converts a JSON value into an AgentTrace:

let json = serde_json::json!([
    { "role": "user", "content": "ping" },
    { "role": "assistant",
      "content": null,
      "tool_calls": [
        { "type": "function",
          "function": { "name": "pong", "arguments": "{}" } }
      ] }
]);

let trace = kanros_agent::parse_openai_style_trace(&json).unwrap();

The parser is forgiving on shape variations but strict on required fields. A missing role or function.name returns AgentError::Format rather than panicking.

Metric functions

The library exposes a small set of pure helpers. Each takes &AgentTrace and returns a primitive number or boolean:

Function	Returns	Meaning
`tool_call_count(&trace)`	`usize`	Total number of tool calls across all assistant turns.
`tool_success_rate(&trace)`	`f64`	Fraction of tool calls that returned a result (vs errored).
`unique_tools_used(&trace)`	`usize`	Distinct tool names invoked.
`redundant_tool_calls(&trace)`	`usize`	Number of identical (name + args) calls beyond the first.
`trace_completed_with_answer(&t)`	`bool`	Final turn is an assistant turn with non-empty `content`.

You compose these into assertions either via type: javascript (read the trace from a vars: entry and call into the helpers via a thin wrapper) or via custom Rust code in a fork.

Scoring patterns

A few patterns we have found useful:

"Did the agent terminate cleanly?"

assert:
  - type: javascript
    code: |
      function evaluate(ctx) {
        const trace = JSON.parse(ctx.vars.trace);
        return trace.turns.at(-1).role === "assistant"
          && trace.turns.at(-1).content.length > 0;
      }

"Did the agent avoid redundant calls?"

Use redundant_tool_calls as a soft cap, not a hard one — some duplication is fine for retry logic.

"Did the agent recover from a tool error?"

Look for a ToolCall with error: Some(_) followed by a successful assistant turn. This pattern is encoded in the upcoming recovered-from-error metric.

Persistence

Both AgentTrace and ToolCall implement Serialize and Deserialize. The runner can persist a trace alongside a cell's CompletionResponse and replay it later for grading without re-running the agent.

Notes

Tool roles are modelled as a distinct TurnRole::Tool rather than a flavour of Assistant. Tool responses are not authored by the model; folding them into the assistant role would make tool_success_rate and friends ambiguous.
A mid-stream snapshot (call requested, not yet resolved) is representable: both result and error are None. Metric implementations treat unresolved calls as neither a success nor a failure.
The kanros-agent API is small on purpose; the goal is for users to compose helpers into their own assertions, not for kanros to ship a thick grading framework.