JavaScript plugins

kanros supports user-authored graders written in JavaScript and run via the embedded boa_engine. They are the smallest possible custom-grader path — no build step, no wasm32 target, no .wit interface. Just a snippet of JavaScript that defines grade(output, context).

The runner lives in kanros-plugin::js. The same engine powers the type: javascript assertion variant.

When to write a JS plugin

• The grader is short — a few lines, no dependencies.
• You are iterating on grader logic and do not want a recompile cycle.
• The logic is a pure function of the eval context, so it stays easy to copy between configs and teams.

For larger graders, graders that need a real library, or graders that need stronger isolation, prefer WASM plugins. The JS runner is in-process and not sandboxed; do not feed it untrusted source.

Shape

A JS plugin is a snippet that defines function grade(output, context) at the top level. The runner sets output (string) and context (the deserialised JSON object) as globals, then evaluates the snippet, then calls grade(output, context).

function grade(output, context) {
  // Pass: the output contains the expected phrase.
  if (output.toLowerCase().includes("key findings")) {
    return { pass: true };
  }
  // Fail with a reason.
  return { pass: false, reason: "missing 'key findings'" };
}

Return-value shapes the runner accepts:

Using a JS grader as an assertion

The javascript assertion variant takes inline code::

assert:
  - type: javascript
    code: |
      function grade(output, context) {
        const summary = JSON.parse(output);
        if (summary.length < 50) {
          return { pass: false, reason: "summary too short" };
        }
        if (!summary.includes(context.vars.expected_phrase)) {
          return { pass: false, reason: "missing expected phrase" };
        }
        return { pass: true };
      }

For longer snippets, load from a file with the templating system:

assert:
  - type: javascript
    code: "{{ file://graders/summary.js }}"

The context object

The runner serialises an AssertionContext-like object and exposes it as the context global. Important fields:

Field names use lowerCamelCase on the JS side (deserialised from the serde-snake_case in kanros-core).

Sandbox / timeout

boa_engine 0.21.1 does not expose a wall-clock interrupt. The runner uses the only soft cap available: a loop iteration limit via RuntimeLimits::set_loop_iteration_limit. We map the user-supplied duration to a coarse iteration budget (about 5 million iterations per second). This catches while (true) {} but is not a hard wall-clock guarantee for code that does not loop.

If you need a stricter cut-off, wrap the assertion call site in tokio::time::timeout on a spawned blocking task — the runner does this for you when assertions exceed runner.timeout_seconds.

Security

JS plugins run in-process and are not sandboxed. They share the runner's address space. They cannot reach the filesystem or network on their own (boa_engine has no Node.js APIs), but the lack of capability isolation means:

• Do not run JS plugins from untrusted sources.
• Do not feed JS plugin source through user-controlled URIs (file:// / https://).
• For untrusted graders, use the WASM plugin path instead.

Determinism

The runner shares one boa_engine::Context across grader calls inside a single JsPluginHost. If you need hermetic isolation per call, build a fresh host. Globals set by one grader are not cleared automatically; keep your graders pure and side-effect-free.

Examples

Score a summary on three criteria

function grade(output, context) {
  const wantPhrases = (context.vars.phrases || []);
  const hits = wantPhrases.filter(p => output.includes(p)).length;
  const score = hits / Math.max(1, wantPhrases.length);
  if (score < 0.5) {
    return { pass: false, score, reason: `hit ${hits}/${wantPhrases.length} required phrases` };
  }
  return { pass: true, score };
}

Validate a structured response

function grade(output, context) {
  let parsed;
  try { parsed = JSON.parse(output); }
  catch (e) { return { pass: false, reason: "output is not JSON" }; }

  if (typeof parsed.title !== "string" || parsed.title.length === 0) {
    return { pass: false, reason: "missing or empty `title`" };
  }
  if (!Array.isArray(parsed.bullets) || parsed.bullets.length < 3) {
    return { pass: false, reason: "expected ≥ 3 bullets" };
  }
  return { pass: true };
}

Notes

• The JS engine is boa_engine, not V8 or QuickJS. Stick to standard ECMAScript; Node-only APIs (require, fs, path, process) are absent.
• The runner expects the snippet to define a top-level grade function. Free-standing expressions are evaluated but ignored.
• For grader logic that needs an LLM judge, use the llm-rubric / factuality / model-graded-closedqa assertions instead. They produce comparable outcomes without a custom JS layer.