Benchmarks

A benchmark runs the same tasks, each pinned to exact commits, under up to six configurations of one agent, so you can compare success rate and cost before changing models or skills.

#A suite

benchmarks/core/suite.yaml
benchmarks/core/tasks/t0.yaml
benchmarks/core/tasks/t1.yaml

# benchmarks/core/suite.yaml
name: core
agent: implement
configurations:
  - { name: sonnet }
  - { name: haiku, agents: [{ agent: implement, model: claude-haiku-5-5 }] }
  - { name: codex, agents: [{ agent: implement, harness: { type: codex } }] }
tasks: [t0, t1]

#A task

# benchmarks/core/tasks/t0.yaml
title: Add a footer link to the pricing page
prompt: "# Add a footer link to the pricing page"
successCriteria:
  - The footer links to /pricing
  - The link test passes
startingRepoRefs:
  - github.com:acme/web@3f2c1a9e5b7d4c6a8e0f1b2c3d4e5f6a7b8c9d0e
sourceRunId: run_abc123
tags: [easy]

#Saving tasks from real work

Any finished run can be saved as a benchmark task. Kiln takes the run's prompt, the exact commit each repository started at, and the work item's acceptance criteria as success criteria (or a default criterion when there are none). It returns the task file for you to add to your suite.

#Running a suite

kiln bench run <suite> runs every task under every configuration, at the pinned commits, without pushing anything. A trial sees history up to its pin only: after checkout Kiln deletes every other branch, tag and remote and prunes the later commits, so the agent cannot read a fix that landed afterwards. The local backend still runs on the host and can read the host's files; use the Docker or Kubernetes backend when the host holds newer clones. Each run is judged against the task's success criteria with the labels success and failure. A task succeeds when the run succeeded and the judge says success.

For each configuration, Kiln reports the number of tasks, the success rate, the average cost and the average duration, and marks whether it is on the Pareto frontier: no other configuration is at least as good on both success rate and cost and better on one.

Benchmarks cost real money. Run them when models, skills or instructions change, not on every commit.

#From results to routing rules

Kiln can suggest routing rules from a suite's results. For each task tag, it takes the cheapest configuration that matches the best success rate for that tag. When that configuration uses a different model than the agent's default, it becomes a rule in a prompt router:

# routers/by-task.yaml
name: by-task
type: prompt
default: claude-sonnet-5-5
routing:
  - { description: easy tasks, model: claude-haiku-5-5 }

For a config-repo factory, Kiln opens this as a pull request labeled kiln:benchmarks. To use it, point the agent at it with model: router:by-task.

#Benchmark runs, trials and results

The benchmarks page and API run suites with more control than kiln bench run:

• Suites and tasks: GET|POST /factories/{id}/benchmark-suites, GET|PATCH|DELETE …/benchmark-suites/{suite}, and …/{suite}/tasks[/{task}]. Adding a task with the same content again is a no-op; another task with the same title is a conflict. POST /runs/{id}/benchmark-task turns a finished run into a task.
• POST …/benchmark-suites/{suite}/runs {configurations, repetitions, concurrency, scorers, tasks} starts a benchmark run. concurrency (1–16, default 1) is how many trials run at the same time; scoring uses the same limit. Each trial is a real agent run, so size it to the host (or workers) and the model provider's rate limits. Each configuration sets, per agent, the harness and model (or a router:<slug>) and the runner. Every task × configuration × repetition is a trial: an agent run of kind benchmark with source BENCHMARK_TRIAL. The run freezes its inputs; it is not idempotent.
• After the trials, each is scored for task correctness and with the scorers you picked. GET /benchmark-runs/{id}/results returns per-configuration averages, an overall score, cost and duration, the Pareto frontier, chart points (quality against cost), spend (execution and judging), coverage and a short narrative, plus a recommendation: highest quality, most efficient (quality per second) and lowest cost.
• GET /benchmark-runs/{id}/trials (filter by task, configuration or agent_run_id) and POST /benchmark-runs/{id}/cancel.

#Related pages

• Scorers
• Replay and canary
• Factory agents
• CLI reference