Evaluate
An eval file is a YAML file in your project's Git repository. It holds one or more eval rubrics, and each rubric defines one thing to check about a run and how to score it. Autoheal applies the rubrics to runs to produce results, such as scores, and the results are what Heal and Benchmark act on.
Eval rubrics answer four questions:
- What do they evaluate? The outcome of a run, or its trajectory.
- Who defines them? Autoheal, for behavior every agent shares and for its builtin agents. You, for your team and your own agents.
- Who evaluates? Code, a decision model, or an LLM judge.
- What are the results for? To score runs, to label runs, or to score benchmark runs.
What eval rubrics evaluate
A rubric checks one of two things, and they fail independently:
| Axis | Question | Typical failures |
|---|---|---|
| Outcome | Did the output (the last agent message) and the artifacts the agent produced meet expectations? | A root cause with no evidence behind it, an incomplete postmortem, a fix that didn't fix the defect |
| Trajectory | Did the agent take the necessary steps for this input? | Queried the wrong data source, ran the same query twice, restarted a pod without asking |
An agent can reach the right answer through a bad path. It can also follow a sound path and write a conclusion the evidence doesn't support. Rubrics for both catch both.
Who defines eval rubrics
Autoheal ships eval rubrics for behavior that every agent shares and for its own builtin agents. You define the rest, specific to your team or to any agent.
Builtin eval rubrics for all agents
Autoheal ships seven eval files that apply to every agent. When post-turn self-improvement is on, every eligible LLM step is scored automatically against all seven. You don't attach them, and attaching one as well scores it a second time.
Five are judged by decision models such as Jev. Two, failed-tool-calls and over-budget, are computed by code from the run.
| Eval file | Axis | Run by | What it scores | Result | If it scores poorly, look at |
|---|---|---|---|---|---|
answer-backed-by-evidence | Outcome | Decision model | Whether the tool results back the main claim of the latest result, and whether the result cites them | no_claim, backed_and_cited, backed_not_cited or not_backed | The agent's instructions or artifact template: cite the evidence id a claim rests on, and state only what a tool result shows |
failed-tool-calls | Trajectory | Code | How often the run's tool calls failed. Reports the failure rate of the worst tool, and of the worst integration | Two failure rates, with the failed and total calls for every tool and integration | The tool or integration that fails most |
why-tool-calls-failed | Trajectory | Decision model | The main cause of the run's failed tool calls | no_failure, agent_error, access_error or service_error | agent_error: the agent. access_error: the integration's owner. service_error: the platform |
instruction-use | Trajectory | Decision model | Whether the agent's instructions fit the task, and whether the agent followed them | Two ratings: relevant, partly_relevant or not_relevant; and followed, partly_followed or not_followed | Instructions that don't fit: the trigger that sent the task here, or guidance the instructions lack. A broken instruction: a rule Heal can make clearer, or move into a step or a Skill |
skill-use | Trajectory | Decision model | How the Skills the agent fetched worked: whether they were relevant, whether the agent followed them, and whether they helped or misled it. Only runs that fetched a Skill are scored | Three ratings: relevance, whether followed, and impact (helped, no_effect or misled) | An irrelevant Skill: its description. An ignored one: the agent's instructions or the Skill's clarity. A misleading one: the Skill's content |
over-budget | Trajectory | Code | The share of turns that cost more than the target in the agent's limits.cost_usd, subagents included. A turn that reports finished work is judged on the total its budget showed: the turns before it, plus the work's cost. Agents with no cost target are not scored | over_budget_rate, with each turn's cost against its target | The agent's cost target in limits.cost_usd, and what drives each over-target turn |
user-follow-up | Outcome | Decision model | The most serious thing the user's follow-up messages said about the agent's answers | no_follow_up, accepted, found_incomplete or corrected | A correction becomes a memory. An incomplete answer changes what the agent's answers must cover |
To score with a builtin by hand, pin a version, or customize one, see Using builtin eval rubrics.
Builtin eval rubrics for builtin agents
Autoheal's builtin agents come with eval rubrics for what a good run of that agent produces. Each is in an eval file that the agent's agent.yaml attaches:
evals:
- evals:regression-attribution
They don't only check correctness. Others check safety, restraint, coverage and whether the agent verified its own work. Each agent has several; this table shows one eval file for each of ten common agents:
| Agent | Eval file | What it scores |
|---|---|---|
| Incident response | mitigation-safety | A proposed mitigation is bounded to one service and revision, names who must approve it, and states the metric that would confirm recovery |
| Alert triage | notification-restraint | The run told the team that had to act, once, and nobody else: nobody for noise, one note to the owner team after a runbook |
| Deploy verification | regression-attribution | Before a movement is called the deploy's, the report rules out a schedule, a dependency and a traffic change, and cites the evidence for each |
| CI/CD failure analyzer | pinged-right-person | The person pinged is the one who had to act: the author of the change, the fallback for a platform failure, or no one for a flaky test |
| Code review | review-signal-to-noise | The review's comments are ones the author would act on: a file, a line and a reason, nits labelled and capped, nothing that restates the diff |
| Feature development | first-push-green | The check runs on the first commit the agent pushed all passed. A red check the agent's own change caused is a miss, whatever its sandbox tests said |
| Vulnerability remediation | remediation-minimality | The proposed change is the smallest one that closes the reachable path, not the newest version, and not a change to services that were never exposed |
| Release readiness | gate-coverage | Each gate the agent declares is answered from evidence gathered in this run, or reported as unanswered |
| Runbook execution | recovery-verification | Before resolving, the run verified recovery on the service's SLO signal over the full window, and escalated after one failed attempt |
| Cloud cost optimization | data-coverage-honesty | Figures are bounded to the window the cost data actually covers, and nothing is extrapolated across a gap |
Eval rubrics you define specific to your team
Autoheal can't know your team's standards, so these eval rubrics don't ship with it. You define them yourself, in eval files in evals/, and attach them to the agents your team owns. You choose the sampling rate, the model and the other fields, so you have full control. Pick a standard that a reviewer on your team already enforces by hand, and write it as a rubric.
These examples show the shape, for four kinds of team:
| Team | Agent | Example eval file | What it checks |
|---|---|---|---|
| Database | Code review | migration-table-context | A review of a schema change states the table's row count and write rate, and cites where each came from, before calling the change safe |
| Infrastructure | Cloud cost optimization | headroom-restraint | A right-sizing proposal declines a saving that would remove headroom an earlier incident showed was needed, and says what proves it |
| AI application | Coding agent spend | model-change-benchmarked | A change to a model, prompt or effort level is benchmarked on the team's own tasks before it is proposed, and reports pass rate, cost and time for every setup tested |
| Web services | Feature development | retry-safety | A change adds or loosens no retry, timeout, backoff or pool setting that would multiply how long a call holds its caller's capacity |
See Writing eval rubrics for the file format and how to attach an eval file to an agent.
Who evaluates
An eval file's type says what scores its rubrics. You can define a rubric of any type:
| Type | Scored by | Good for | Sampling |
|---|---|---|---|
script | Code. The result is deterministic | Counts and rates computed from the run, such as failed tool calls or cost over target | Every run |
decision_judge | A decision model such as Jev, which answers each rubric by choosing a label, and writes no text | Quick checks on every turn: was the claim cited, were the instructions followed | Default 0.5 |
llm_judge | An LLM, which reads the conversation and writes a reasoned answer for each rubric | Deeper questions, and numeric scores | Default 0.5 |
A decision rubric is faster and cheaper than an LLM judge rubric, but it is boolean or categorical, never numeric. When a deployment has no decision model set up, the judge model answers a decision eval file instead.
judge and decision are the old names of llm_judge and decision_judge. An eval file that still says type: judge or type: decision is scored under the new name, and the Git registry shows a warning to rename it.
Coming soon: judges will be agents that read the trace. A judge will read only the messages that settle each rubric, and cite them, so a run longer than the judge's context window can still be scored. looks_for will also narrow a judge to the matching turns.
When eval rubrics run
Automatically
When a conversation has been quiet for 10 minutes after a successful turn, Autoheal scores it with every attached eval file: each judge or decision at its sampling rate, each script every time. A conversation that keeps going is scored once, not after every turn.
By hand
Select Score on a run page and pick up to 20 eval files. Sampling doesn't apply, so every eval file you pick scores the run. You can score with an eval file from another branch or an older commit, which is how you test a new rubric on runs that already happened, without re-running the agent.
In agents with several LLM steps
Each step is its own conversation, scored with the eval files on that step:
steps:
draft:
instructions: Draft the incident summary.
evals: [evals:evidence-quality]
polish:
instructions: Tighten the summary.
with:
prompt: ${{ steps.draft.output }}
evals: [evals:evidence-quality, ./evals:tone]
A run of this agent gets three scores, one per step and eval file. A step inside a while loop is scored once per iteration: the Scores table lists every iteration, and the Evals page's cards count each run's last iteration. The Evals page groups the scores by step.
Paused, failed and cancelled LLM steps, and subagents' runs, aren't scored automatically. An LLM step that succeeded is scored even when a later step of the run fails.
What eval results are for
| Use | Applies to | Looks at | Example |
|---|---|---|---|
| Score runs | Any finished run of the agent | The output, or how the agent got there | "Did the root cause cite evidence from a tool result?" |
| Label runs (coming soon) | Any run, finished or not | The input, the output or the trajectory | "Is this a db_feature, an app_feature or an infra_fix?" |
| Score benchmark runs | Benchmark cases | The output or trajectory, compared with the case's known answer when it has one | "Does the root-cause artifact name resource contention on payments-db?" |
Score runs
A score says how good or bad a run was. Each rubric sets its own scoring: a number between a minimum and a maximum (numeric), true or false (boolean), or one label from a fixed list (categorical). A script eval file reports its own measures instead, such as a failure rate.
Scores from the builtin and team eval rubrics above are what Heal starts from. See Reading scores for where they show up.
Label runs (coming soon)
Coming soon. Labeling is not available yet.
A labeling rubric assigns one label to each run based on the kind or complexity of its input, its output or its trajectory. For a QA agent, the labels could be db_feature, app_feature and infra_fix. Use separate rubrics for different traits. A labeling rubric uses the same eval file and scoring path as other rubrics, but returns a label instead of a grade. Other examples include:
- incidents for your highest-tier customers;
- runs that finished without writing an artifact;
- runs where a person corrected the agent;
- investigations of database incidents versus deploy incidents.
Labels make aggregate scores useful. An overall root-cause score of 0.82 can hide 0.95 on deploy incidents and 0.4 on database incidents. Heal looks for exactly that kind of split.
Score benchmark runs
A benchmark rubric compares a run with a reference answer that a person approved, such as the root cause the team confirmed for a past incident. It runs only on benchmark cases. A case with no reference answer is still scored: the rubric judges the run on its own terms. A rule can be as simple as "the root-cause artifact names payments-db and resource contention", and the result is pass or fail.
Reading scores
What a result records
Each result records the agent version, the eval version, the branch and the run it scored. Scores from two versions of a rubric are never mixed in the same chart.
The Evals page summarizes scores by agent version, eval version, rubric, branch and day.
Querying and comparing scores
Self-improvement agents will be able to query eval scores with SQL, including comparisons between two sets of runs. Heal uses this to find poor scores and the groups of runs where they cluster.
A comparison page will put the scores of any two sets of runs side by side:
- runs in a time window, or a list of run ids, on any branch;
- the runs of a benchmark execution on any branch.
Writing eval rubrics
The eval file
This eval file holds one judge rubric and scores an alert-triage agent:
name: Root cause grouping
description: >-
Alerts that share a cause are one group, linked through a cited dependency edge;
a service taking retries is linked but not blamed.
type: llm_judge
sampling:
rate: 1.0
rubrics:
- name: grouped
prompt: >-
What fraction of the co-firing alerts are placed correctly, in the cause's group
with the edge that links them cited, or in a group of their own when nothing links
them?
scoring:
type: numeric
min: 0
max: 1
| Field | What it does |
|---|---|
type | llm_judge, decision_judge or script. See Who evaluates. |
rubrics | The eval rubrics in the file. Each has a name, a prompt and its own scoring. A decision rubric is boolean or categorical with 2 to 255 labels, not numeric. |
looks_for | The tools and bash commands the eval file is about, such as {tools: [restart_pod]}. A run with none of them is marked not applicable, and no model is asked. |
applies_to | Score only runs that saved this kind of artifact, such as hypothesis. Other runs are marked not applicable. |
model | LLM judge eval files only: the judge model, written the same way as an agent's model, such as claude-sonnet-5. If your deployment doesn't offer that model, the push shows an error that lists the models it does offer. Without it, the environment's default judge model is used. A decision eval file can't set it. |
sampling | For LLM judge and decision eval files: what fraction of runs to score automatically. Default is 0.5. Scoring by hand ignores it, and a script eval file scores every run whatever sampling says. |
Where eval files live
| Path | Who can use it | Reference |
|---|---|---|
evals/<name>.yaml | Any agent in the project | evals:<name> |
agents/<agent>/evals/<name>.yaml | Only that agent | ./evals:<name> |
| Published by Autoheal | Any agent, without copying | builtin/evals:<name> |
An eval file in the agent's folder is part of the agent: editing it creates a new agent version too. Editing a project eval file creates only a new eval version.
Attaching eval files to an agent
An agent attaches eval files in its agent.yaml:
evals:
- evals:evidence-quality
- {eval: ./evals:asked-before-restart, auto: false}
A plain reference scores automatically. auto: false keeps the eval file for manual scoring only. It is the way to stop a script eval file from scoring automatically, because scripts ignore sampling.
Using builtin eval rubrics
Scoring by hand. To score a run with a builtin eval file yourself, attach the published eval file to the agent without copying it:
evals:
- builtin/evals:user-follow-up
For manual-only scoring, use {eval: builtin/evals:user-follow-up, auto: false} instead of the string attachment. Automatic scoring defaults to on.
Versions. Builtin eval files use the current deployed content for every new manual, automatic, reuse or benchmark request. A result records the builtin address and the content digest, with no eval commit, so branch and commit choices don't apply. Retries of a prepared request keep that request's definition. Project eval files keep their existing commit and version behavior.
Inheritance and customizing. Agents that extend a builtin inherit its private eval attachments. Private builtin eval files cannot be referenced directly. To customize a published eval file, copy its YAML into your project's evals/ folder and attach the project copy.
Changing eval rubrics
Eval rubrics are files, so you change them like code: on a branch, reviewed in a pull request. Heal proposes rubric changes the same way, when it finds a behavior no rubric watches.
To check that a new rubric measures what you meant, score a few past runs with it from its branch before you merge. Pick some runs that should pass and some that should fail.
Limits
- Only single-LLM agents can be scored by hand. Score on a run page and benchmarks score one conversation. An agent with several LLM steps is scored automatically, per step.
- A loop inside a loop keeps only the last outer pass. An LLM step's iterations are numbered within its innermost loop, so each pass of the outer loop reuses them. Conversations from earlier outer passes aren't scored, and the step's results keep only the last outer pass.
- A judge is a model. Its verdicts can be wrong. Read the reasoning on a few results before you rely on a new rubric.
Related
- Observe: the trace a judge reads
- Heal: what happens after weak or unusual scores
- Benchmark: where benchmark rubrics run
- Capabilities: which agents can run and read evals