Scorers

A scorer is a rubric in scorers/<name>/scorer.md that grades a share of an agent's finished runs with one of a fixed set of labels.

#A scorer file

---
name: acceptance-criteria
description: Checks whether the change meets its acceptance criteria
agents: [implement]
labels:
  - { value: meets, score: 1, description: Every criterion has a passing test }
  - { value: partly, score: 0.5 }
  - { value: misses, score: 0 }
passingScore: 0.8
samplingRate: 25
model: claude-haiku-5-5
selfImprovement: true
---
Pass "meets" only if every acceptance criterion has a passing test or evidence.
Use "partly" when some criteria are covered. Use "misses" when a criterion has no evidence.

The body is the rubric the judge reads.

#Which runs are scored

When a work run finishes, Kiln scores it with every scorer that lists its agent.

• Sampling is by a stable hash of the run ID and the scorer name, so the same run is always in or out of the sample for that scorer. About samplingRate percent of runs are scored.
• Replays and benchmark runs ignore sampling: when Kiln scores them, every scorer for the agent grades them.
• The server can be set to score every run, ignoring samplingRate.
• A run is scored at most once per scorer.

#How a run is judged

The judge gets the rubric, the run's summary, its result file, the last part of its event transcript and the agent's name, and picks one label.

By default the judge is a typed decision: the labels become the options of one choice question, so judging is cheap enough to score many more runs. When the top label's probability is below 0.7 and an LLM judge is configured, the run goes to the LLM judge instead.

A label that is not in the scorer's list is not recorded; Kiln writes an invalid_label entry to the audit log.

#Scores

Each score records the run, factory, agent, scorer, label, numeric score, pass or fail, the judge's rationale, which judge decided, and the definition SHA the run used. Scores from work runs of a scorer with selfImprovement: true count toward self-improvement until a proposal marks them reviewed.

Use scores in:

• Dashboard metrics: pass rate per agent and per definition,
• Self-improvement: unreviewed failures trigger a proposal,
• Replay and canary: pass rates before and after a change.

#Default scorers

A factory made with the setup wizard starts with five scorers, each with a written rubric you can edit: Verbosity, Task Compliance, Procedure Compliance, Efficiency and Code Quality (implement agent only). Task Compliance, Procedure Compliance and Code Quality feed self-improvement. All five pass at a score of 0.5 and grade 10 percent of runs.

The Kiln factory itself (factories/kiln-self) ships the same five. A sixth, Task Correctness (kind: benchmark_task_correctness), grades benchmark trials against their success criteria and never scores everyday runs.

Two more scorer keys: agents entries may be { name, includeDescendants: true } to include a run's subagents as evidence, and runner, secrets and mcpServers set up the judge.

#Scorers API

• GET|POST /factories/{id}/scorers, GET|PATCH|DELETE /factories/{id}/scorers/{slug}: the UI's scorer editor. Writes go into scorers/<slug>/scorer.md (a pull request for config-repo factories). Each update bumps the scorer's version; deleting removes its scores too. Creating a scorer does not score past runs.
• GET …/scorers/{slug}/results (with include_reasons=true, run_id=) and GET …/scorers/{slug}/metrics (pass rate and series; include_cost=true adds judge cost).
• Manual scoring, which ignores sampling and replaces earlier scores: POST …/scorers/{slug}/dispatch {runIds}, POST /factories/{id}/runs/{runId}/scorer-dispatch {scorers}, or POST /runs/{id}/score for every scorer that covers the run's agent.
• GET /runs/{id}/scores lists a run's scores with triggerSource (automatic or manual).

#Related pages

• Self-improvement
• Dashboard metrics
• Benchmarks
• Typed decisions