Troubleshooting

This page collects the failure modes that come up most often when running kanros against real provider endpoints, real caches, and real configs. Each entry pairs the user-visible symptom with the diagnostic command that confirms the root cause, plus the fix.

If you hit a problem that isn't covered here, please email support@kanros.dev with the diagnostic output — we'd rather extend this page than have you debug it twice.

kanros run hangs (no output, no progress)

Symptom. kanros run -c kanros.yaml prints the run header but never emits a cell result. No timeout fires, no error surfaces. Ctrl-C exits cleanly, so the process isn't deadlocked — it's blocked on I/O.

In nearly every case this is the configured provider's network call sitting in a TCP retry loop because the endpoint, the API key, or the local DNS is wrong.

Diagnose.

# Re-run with the structured tracing layer cranked up. The provider
# emits a span around every outbound HTTPS call; if you see the span
# open but never close, the provider is the culprit. `--concurrency 1`
# overrides runner.max_concurrency so cells run one at a time.
RUST_LOG=kanros_providers=debug,kanros_runner=info \
    kanros run -c kanros.yaml --concurrency 1

If the trace shows the request going out but no response, hit the provider's health endpoint directly:

# Substitute the base URL from your provider config.
curl --max-time 5 -sS -o /dev/null -w '%{http_code}\n' \
    https://api.openai.com/v1/models \
    -H "Authorization: Bearer $OPENAI_API_KEY"

A non-2xx (or a curl timeout) confirms the issue is upstream of kanros.

Fix. Verify the provider.config.base_url field in your YAML, check that the API key has not been rotated, and confirm no corporate proxy is intercepting outbound HTTPS. For local providers (ollama, llama-cpp, lm-studio) confirm the daemon is up on the configured port.

Cache permission denied

Symptom. kanros run aborts during cache initialization with Error: Permission denied (os error 13) and a path under ~/.cache/kanros/. This typically happens after running kanros under sudo once: the cache directory then ends up owned by root, and subsequent un-privileged runs cannot open the SQLite file.

Diagnose.

# Inspect ownership and permissions on the cache root.
ls -ldn ~/.cache/kanros ~/.cache/kanros/*.sqlite 2>/dev/null

A line showing UID 0 (root) when you're running as a regular user is the giveaway.

Fix. Either chown the directory back to your user, or wipe the cache and let kanros recreate it with the correct mode:

sudo chown -R "$(id -u):$(id -g)" ~/.cache/kanros
# — or —
rm -rf ~/.cache/kanros && kanros cache info

The second form is preferred when you don't know what else might be inside; kanros cache info rebuilds the directory at the correct mode (0700) and seeds an empty SQLite file.

Schema validation fails on valid YAML

Symptom. A kanros.yaml that worked yesterday now fails with additionalProperties: ... or unknown variant ..., even though the file is unchanged and parses as valid YAML in your editor.

This is almost always schema drift: you upgraded kanros (or built from a newer commit) and the embedded JSON Schema picked up a new required field, or renamed a variant of AssertionType.

Diagnose.

# Print the schema your installed binary believes in, and validate
# the config against it explicitly.
kanros schema > /tmp/kanros.schema.json
kanros validate --config kanros.yaml

validate reports the field and line at which parsing failed. Cross-reference against the live schema (kanros schema) and the Configuration reference.

Note that validate checks the config's own structure plus the generation parameters (temperature, top_p, max_tokens, seed, stop, response_schema, response_format) on each provider. The rest of the per-provider config: block is passed through to the provider crate, so a bad key there (e.g. endpoint instead of base_url), a missing API key, or an unset env:NAME credential surfaces when the provider is built at the start of kanros run (exit code 2), not during validate.

Fix. Either pin to the previous kanros release until you've migrated the config, or regenerate any tooling (IDE plugins, CI lints) that consumes the schema:

kanros schema > tooling/kanros.schema.json
git add tooling/kanros.schema.json

If you maintain a fork of the config schema, regenerating against the new binary is the canonical fix — kanros's schema is the source of truth.

API key not found (keyring lookup)

Symptom. Provider initialization fails with Error: no API key for provider 'openai' even though OPENAI_API_KEY is set in your shell, or vice versa: the env var is unset but you expect kanros to read from the OS keychain.

kanros resolves credentials in a fixed order, and kanros auth writes to the OS keychain (Keychain on macOS, Secret Service on Linux, Windows Credential Manager on Windows). The lookup order is:

  1. The explicit api_key field in the provider config. Use api_key: env:NAME to read a specific environment variable; if NAME is unset the run stops with an error instead of falling through to steps 2 and 3.
  2. The provider-specific env var (OPENAI_API_KEY, ANTHROPIC_API_KEY, …).
  3. The OS keychain, queried via the keyring crate under the service name kanros and an account named after the provider id.

If step 2 wins, step 3 is never consulted — even if the env var is empty-string.

Diagnose.

# List the providers that have a key stored in the kanros keyring.
kanros auth list
# Check whether a key is stored for one provider ("not set" if absent).
kanros auth get openai
# Check whether the env var is exported to child processes.
env | grep -c '^OPENAI_API_KEY='

Fix.

  • If you intended to use the keyring, unset the env var:
    unset OPENAI_API_KEY
    kanros auth set openai
    
  • If you intended to use the env var, make sure it's exported, not just set in the current shell scope:
    export OPENAI_API_KEY="sk-..."
    
  • If kanros auth list does not show the provider even after you ran kanros auth set, the OS keychain itself rejected the write. On headless Linux this usually means secret-service isn't running — start gnome-keyring-daemon --components=secrets or fall back to env vars.

An assertion reports error instead of passing

Symptom. A cell shows [ERR ] with a line such as wasm errored: … the plugin host is disabled, perplexity errored: provider … did not return logprobs, moderation errored: provider … doesn't support moderation, or classifier errored: classifier assertion needs HUGGINGFACE_API_KEY, and the run exits 1.

kanros never counts an assertion it could not evaluate as a pass. Common causes and fixes:

MessageFix
plugin host is disabledRe-run with kanros run --plugins.
plugin not foundFix the wasm:// path in the plugin.source field.
did not return logprobs (perplexity)perplexity works only with the openai provider.
doesn't support moderationPoint the moderation assertion's provider: at openai.
needs HUGGINGFACE_API_KEY (classifier)Export HUGGINGFACE_API_KEY (or KANROS_HUGGINGFACE_API_KEY).
grader returned malformed JSON (llm-rubric)The grader provider: is not a real model (e.g. echo). Configure one, or pass --grader to kanros redteam generate.

A cost assertion reports skipped (not error) when the provider has no known price; skipped assertions do not fail the run.

See also