Comparison

kanros vs promptfoo: the independent alternative

promptfoo is an excellent, widely adopted tool — and since March 2026 it is being acquired by OpenAI. If your evaluation and red-teaming stack should stay independent of the model vendors it tests, kanros is built for exactly that: a fast Rust CLI, EU-based, local-first, with a one-command migration from your existing promptfoo configs.

This page is written by the kanros team, so read it as a vendor comparison. Every claim about promptfoo links to public sources, we call out where promptfoo is genuinely stronger, and we keep pricing and facts dated. Last updated: October 17, 2026.

At a glance

promptfoo details are drawn from promptfoo.dev and its GitHub repository as of October 17, 2026. Spotted something out of date? Email support@kanros.dev.

Why teams pick kanros

• Independence. Your red-team tool reports vulnerabilities in the models you buy. After the acquisition, promptfoo is owned by one of those model vendors. kanros is an independent European company — the test results answer to you, not to a provider's roadmap.
• One binary, no runtime. kanros ships as a single static binary for macOS, Linux, and Windows. No Node.js, no npm tree in your CI image — cold-start in CI is checksum, extract, run.
• Determinism and diffing. A deterministic core, a response cache, seeded red-team probes, and kanros diff make "did this prompt change break anything?" a reviewable, reproducible answer instead of a vibe.
• Sandboxed custom logic. Custom graders run as WASM plugins with a default-deny sandbox (no filesystem, network, or clock unless allowed). Untrusted grader code can't exfiltrate your outputs.
• CI-native security output. SARIF for code scanning, JUnit for test dashboards, GitHub annotations for PRs — findings land in the tools your team already reviews.
• EU footing. Built in Germany. No telemetry in the binary, and the optional Kanros Cloud is hosted in the EU (Azure, Italy North) with a results-only, keys-stay-with-you model.

Where promptfoo is stronger

• Ecosystem and community. promptfoo reports hundreds of thousands of developers and very wide enterprise adoption. Its catalog of integrations, examples, and community answers is larger than ours today.
• Red-team breadth. Its 50+ attack plugins with OWASP LLM Top-10, NIST AI RMF, and MITRE ATLAS presets are the broadest public catalog. kanros covers the core strategy families and is closing the gap release by release.
• Node-native graders. If your custom assertions lean on Node.js built-ins, promptfoo runs them as-is; kanros deliberately sandboxes custom code, which means porting.

Migrate in one command

kanros migrate reads your promptfooconfig.yaml and emits a kanros config — renaming fields, resolving $ref assertion templates, rewriting JavaScript expression assertions, and flagging anything that needs human review with TODO comments instead of failing:

$ kanros migrate promptfooconfig.yaml
$ kanros validate -c kanros.yaml
$ kanros run -c kanros.yaml

What migrates automatically

• providers:, including the config: block: openai:chat:gpt-4o-mini becomes openai:gpt-4o-mini, sampling settings move to the provider entry, and '{{ env.NAME }}' becomes env:NAME.
• prompts:, tests: (inline lists and file:// CSV, JSONL, YAML, JSON references), vars:, and defaultTest.
• Common assertions: equals, contains, regex, javascript expressions, llm-rubric, factuality, similar, latency, cost, levenshtein, and weight: (converted to a weighted assertion set).

What needs manual review

• JavaScript graders that use Node.js built-ins (fs, http, …). kanros runs JavaScript in an embedded engine without Node APIs; port these to a pure function or a WASM plugin.
• Inline Python assertions and custom scoring functions. They are dropped with a TODO; file-based Python paths are kept.
• Model-graded assertions without an explicit grader. promptfoo picks a default grader; the migrator uses your first provider and adds a TODO so you choose deliberately.
• Glob test paths, script or spreadsheet test sources, provider config keys kanros does not support, and the redteam: block (renamed best-effort, flagged for review).

Every item above is reported as a # TODO(migrate): line at the top of the output, and the command prints how many there are. The migration docs list each mapping.

Benchmark: 1,000 test cases

An outside tester ran the same 1,000-case suite through both tools with an offline, deterministic provider, so the numbers measure the tools, not model latency:

Much of promptfoo's time is Node.js start-up and module loading, so the gap is largest for short suites and in CI jobs that start a fresh process each time. With real model calls, provider latency dominates both tools.

Reproduce it with the harness in the kanros repository. It measures median wall-clock, peak memory and start-up time over N runs and writes results.json. The committed suite has 30 cases; on that suite we measured 10 ms vs 1,550 ms.

$ cd benches/vs-promptfoo
$ N=10 ./run.sh
$ cat results.json

Methodology and the full results are on the benchmarks page.

FAQ