Assertions

An assertion grades a model output. Every assertion the user can write in YAML is a variant of the AssertionSpec enum in the frozen kanros-core trait layer — run kanros schema for the full machine-readable catalogue. The runner turns each spec into a concrete Assertion and records the outcome (Pass, Fail, Error, Skipped) per cell.

YAML always uses type: as the discriminator and kebab-case for the variant name:

assert:
  - type: contains
    value: "key findings"

Outcome states

A run exit code is 0 only when every cell's every assertion is pass or skipped. An assertion that cannot be evaluated is never counted as a pass: it reports error, which fails the run.

Assertions listed under a config's default_test.assert are appended to every test's own assert list, and default_test.vars fill in any variable a test leaves unset.

Catalogue

The full list of AssertionSpec variants below is grouped by family. Every name is the exact YAML type: value.

Deterministic

• equals — exact string equality. Takes value: String.
• not-equals — inverse.
• contains — case-sensitive substring match.
• icontains — case-insensitive substring match.
• not-contains — substring must not appear.
• starts-with — required prefix.
• ends-with — required suffix.
• regex — Rust regex syntax. Takes pattern: String.
• not-regex — inverse.
• contains-all — every needle in values: [String] must appear.
• contains-any — at least one needle must appear.
• contains-none — none of the needles may appear.
• contains-json — output must contain a JSON object/array somewhere.
• is-json — output is itself valid JSON.
• json-schema — output is valid JSON matching the supplied draft 2020-12 schema: value.
• is-sql — output parses as SQL in one of the supported dialects.
• is-xml — output is well-formed XML.
• length — character length in [min, max] (either bound optional).
• word-count — word count in [min, max].

Performance

• cost — USD cost must not exceed max_usd: f64. Evaluated from token usage and kanros' built-in price table; reports skipped when the provider/model has no known price (local models, echo, unlisted models).
• latency — wall-clock latency must not exceed max_ms: u64.
• perplexity — output perplexity must not exceed max: f64. Needs token logprobs, so it works only with the openai provider (chat completions logprobs); on any other provider it reports error.

Custom code

• javascript — runs code: String via the embedded boa_engine. The source must define function evaluate(ctx) returning truthy/falsy.
• python — runs a Python script: via subprocess.
• webhook — POSTs the context to url: String; the response decides pass/fail.
• wasm — calls a WASM (Component Model) plugin. Takes plugin: { source: "wasm://path/to/plugin.wasm" } and an optional config: JSON value. Alias of custom.

Similarity

• similarity — cosine similarity of output and reference embeddings. Requires reference: String, threshold: f32, and embedder: ProviderRef.
• levenshtein — edit distance against a reference: String, bounded by max_distance: u64.

Model-graded

• llm-rubric — LLM-as-judge with a free-text rubric. Takes value: String (the rubric) and provider: ProviderRef (the grader model).
• factuality — factuality check against a reference: answer.
• model-graded-closedqa — closed-QA model-graded check.
• answer-relevance — answer-relevance check.

RAG metrics

See RAG metrics for the full semantics.

• context-recall — recall over a reference: answer with threshold: f32 and grader provider:.
• context-precision — precision over the retrieved context.
• context-relevance — relevance of the retrieved context to the question.
• faithfulness — every claim in the answer is supported by the supplied context:.

NLP metrics

• bleu — BLEU score against a reference:, bounded by threshold:.
• rouge-n — ROUGE-N score; takes n: u8 (typically 1 or 2).
• meteor — METEOR score against a reference:.

Safety

• moderation — runs a moderation API; pass if no categories are flagged. Optional categories: whitelist restricts which flags matter. The provider: must expose a moderation endpoint (for example openai); pointing it at one that does not reports error.
• classifier — runs a HuggingFace classifier; pass if label: is below threshold:. Needs HUGGINGFACE_API_KEY (or KANROS_HUGGINGFACE_API_KEY); without it the assertion reports error.

Composite

• assert-set — combine N child assertions: with op: and | or | weighted and optional threshold:.
• select-best — rank N model outputs by rubric: and pass only the best.

Plugin

• custom — call a user-provided WASM assertion plugin via a plugin: PluginRef (wasm is an alias). The plugin host is off by default: without kanros run --plugins, or when the plugin file cannot be found, the assertion reports error — it never passes silently.

assert:
  - type: custom
    plugin: { source: "wasm://plugins/my_assertion.wasm" }
    config: { threshold: 0.8 }

Any assertion type that is accepted by the schema but not implemented by the running binary also reports error rather than passing.

Worked examples

A composite: pass only if every child passes.

assert:
  - type: assert-set
    op: and
    assertions:
      - type: contains
        value: "summary"
      - type: latency
        max_ms: 1000
      - type: word-count
        min: 50
        max: 250

A weighted composite with a score threshold:

assert:
  - type: assert-set
    op: weighted
    threshold: 0.7
    assertions:
      - type: similarity
        reference: "Apollo 11 landed in 1969."
        threshold: 0.6
        embedder: { id: openai:text-embedding-3-small }
      - type: contains
        value: "1969"

A model-graded rubric (one grader call per cell):

assert:
  - type: llm-rubric
    value: |
      The summary must mention the launch date, the mission lead, and
      the planned next step. Style must be neutral.
    provider:
      id: openai:gpt-4o

A JSON-schema check:

assert:
  - type: json-schema
    schema:
      type: object
      required: [summary, key_findings]
      properties:
        summary: { type: string, minLength: 10 }
        key_findings:
          type: array
          items: { type: string }
          minItems: 1

Notes

• Custom code assertions (javascript, python, webhook, wasm) run inside the runner process by default; for stronger isolation prefer the wasm assertion which uses the capability-sandboxed WASM plugin host described in WASM plugins.
• Each assertion records both its outcome and (where applicable) a numeric score in [0.0, 1.0]. Composite assertions read these scores to compute their own verdict.