RAG metrics

Retrieval-augmented generation (RAG) systems need their own grading vocabulary: a generic contains assertion cannot tell you whether the model invented a fact that was not in the retrieved context. kanros ships two complementary RAG APIs in the kanros-rag crate.

The two APIs

Assertion-style holistic grading (metrics)

One judge LLM call per metric. Returns an AssertionOutcome directly, and is what the assertion engine wires up for the context-recall, context-precision, context-relevance, and faithfulness variants.

This is the path you reach for when you want a normal type: context-recall line in your kanros.yaml.

Ragas-style decomposed grading (ragas)

Multiple judge LLM calls — one per sentence or claim — for fine-grained scoring that mirrors the reference Ragas metrics. Returns a plain f64 in [0.0, 1.0]. Useful when you want to drive the metric from Rust code, log the intermediate per-sentence verdicts, or implement your own assertion shape on top.

Both APIs share semantics for the four core metrics.

The four metrics

Context recall

Of all the claims in the ground truth answer, what fraction are covered by the retrieved context?

Higher recall means the retriever is doing its job — it pulls back enough context that a downstream model could answer correctly.

assert:
  - type: context-recall
    reference: "Apollo 11 landed on the Moon on 20 July 1969."
    threshold: 0.8
    provider: { id: openai:gpt-4o }

Context precision

Of all the claims in the model's answer, what fraction are supported by the retrieved context?

Low precision means the generator is wandering off-context — making claims the retriever did not back up.

assert:
  - type: context-precision
    reference: "Apollo 11 landed on 20 July 1969."
    threshold: 0.9
    provider: { id: openai:gpt-4o }

Context relevance

How on-topic is the retrieved context, treating it as candidate evidence for the question? Sentences that bear no relation to the question penalise the score.

assert:
  - type: context-relevance
    threshold: 0.7
    provider: { id: openai:gpt-4o }

Faithfulness

Every claim in the model's answer must be entailed by the retrieved context. Faithfulness is the strictest of the four — even a true claim that is not in the context fails.

assert:
  - type: faithfulness
    context: |
      Apollo 11 launched on 16 July 1969 and landed on the Moon on
      20 July 1969. Neil Armstrong was the mission commander.
    threshold: 1.0
    provider: { id: openai:gpt-4o }

How a metric runs

For the assertion-style metrics API, the judge LLM is asked to return:

{ "score": 0.0, "reason": "..." }

…wrapped in arbitrary prose. kanros locates the first balanced {…} block in the response (find_json_island) and parses it. The score is compared to threshold; the reason is reported on failure.

For the Ragas-style API, the judge answers a sequence of yes/no questions — one per sentence in the ground truth, one per claim in the answer, and so on. Per-item verdicts are aggregated into the final score. Individual provider errors are absorbed (the affected item is counted as "not supported"), so a single flaky judge call does not crash the whole metric.

Choosing a grader

The four assertion variants take a provider: block exactly like any other model-graded assertion. The grader can be a different family from the system under test:

providers:
  - id: openai:gpt-4o-mini   # system under test (cheap)

tests:
  - vars: { question: "When did Apollo 11 land?" }
    assert:
      - type: faithfulness
        context: "Apollo 11 landed on 20 July 1969."
        threshold: 0.9
        provider:
          id: anthropic:claude-3-5-sonnet   # grader (more capable)

Notes

  • These metrics call the grader per cell. Budget accordingly — a 4-provider × 50-test suite with all four metrics is 800 grader calls.
  • The grader prompts and JSON-extraction heuristic are duplicated between kanros-rag::metrics and kanros-assertions::families::model_graded rather than coupled. They are kept identical; update both when changing.
  • The ragas::* functions absorb individual judge errors. They never panic on a malformed judge response.