Vision input

Some kanros providers (OpenAI's gpt-4o, gpt-4-turbo, Anthropic's Claude 3+, and others) accept multimodal input — text and images in the same prompt. kanros surfaces vision through a simple template marker that any prompt can embed:

providers:
  - id: openai:gpt-4o-mini
prompts:
  - |
    Describe what you see in this image:
    {{ image:fixtures/cat.png }}
tests:
  - vars: {}
    assert:
      - type: icontains
        value: cat

Prompts are plain strings (or file:// / https:// URIs); the {{ image:PATH }} marker can appear anywhere inside one, and a prompt may carry more than one marker. The marker is removed from the text that is sent, and each referenced file becomes an image attachment on the same request.

At runtime the marker is recognised by [kanros_core::preprocess_vision_template], the file is read from disk, base64-encoded, and attached to the CompletionRequest as a VisionContent block. The provider then translates that into the correct wire format:

Capability gating

Providers that don't support vision (echo, ollama with text-only models, Anthropic Claude 2.x, OpenAI gpt-3.5-turbo, ...) return ProviderError::VisionUnsupported when a vision payload is present. The runner surfaces this as a clean per-cell error rather than silently dropping the image.

Path resolution

Relative paths in {{ image:PATH }} are resolved against the directory containing the kanros config, not the current working directory (so fixtures/cat.png always means "the fixtures/ folder next to the YAML file"). Absolute paths are used unchanged. A missing file fails the cell with an error; it is never silently dropped.

Supported MIME types

The MIME type is inferred from the file extension:

• .png — image/png
• .jpg, .jpeg — image/jpeg
• .gif — image/gif
• .webp — image/webp

Other extensions return VisionError::UnknownMime at render time.

Programmatic API

use kanros_core::{CompletionRequest, VisionContent, ImageData};

let req = CompletionRequest::user_with_vision(
    "what is in this image?",
    vec![VisionContent {
        url_or_data: ImageData::Url("https://example.com/cat.png".into()),
        mime: "image/png".into(),
    }],
);

ImageData::Url(...) is forwarded verbatim to the provider (which fetches the URL itself); ImageData::Base64(...) is inlined.