Skip to main content

Evaluate

An eval file is a YAML file in your project's Git repository. It holds one or more eval rubrics, and each rubric defines one thing to check about a run and how to score it. Autoheal applies the rubrics to runs to produce results, such as scores, and the results are what Heal and Benchmark act on.

Eval rubrics answer four questions:

  1. What do they evaluate? The outcome of a run, or its trajectory.
  2. Who defines them? Autoheal, for behavior every agent shares and for its builtin agents. You, for your team and your own agents.
  3. Who evaluates? Code, a decision model, or an LLM judge.
  4. What are the results for? To score runs, to label runs, or to score benchmark runs.

What eval rubrics evaluate​

A rubric checks one of two things, and they fail independently:

AxisQuestionTypical failures
OutcomeDid the output (the last agent message) and the artifacts the agent produced meet expectations?A root cause with no evidence behind it, an incomplete postmortem, a fix that didn't fix the defect
TrajectoryDid the agent take the necessary steps for this input?Queried the wrong data source, ran the same query twice, restarted a pod without asking

An agent can reach the right answer through a bad path. It can also follow a sound path and write a conclusion the evidence doesn't support. Rubrics for both catch both.

Who defines eval rubrics​

Autoheal ships eval rubrics for behavior that every agent shares and for its own builtin agents. You define the rest, specific to your team or to any agent.

Builtin eval rubrics for all agents​

Autoheal ships seven eval files that apply to every agent. When post-turn self-improvement is on, every eligible LLM step is scored automatically against all seven. You don't attach them, and attaching one as well scores it a second time.

Five are judged by decision models such as Jev. Two, failed-tool-calls and over-budget, are computed by code from the run.

Eval fileAxisRun byWhat it scoresResultIf it scores poorly, look at
answer-backed-by-evidenceOutcomeDecision modelWhether the tool results back the main claim of the latest result, and whether the result cites themno_claim, backed_and_cited, backed_not_cited or not_backedThe agent's instructions or artifact template: cite the evidence id a claim rests on, and state only what a tool result shows
failed-tool-callsTrajectoryCodeHow often the run's tool calls failed. Reports the failure rate of the worst tool, and of the worst integrationTwo failure rates, with the failed and total calls for every tool and integrationThe tool or integration that fails most
why-tool-calls-failedTrajectoryDecision modelThe main cause of the run's failed tool callsno_failure, agent_error, access_error or service_erroragent_error: the agent. access_error: the integration's owner. service_error: the platform
instruction-useTrajectoryDecision modelWhether the agent's instructions fit the task, and whether the agent followed themTwo ratings: relevant, partly_relevant or not_relevant; and followed, partly_followed or not_followedInstructions that don't fit: the trigger that sent the task here, or guidance the instructions lack. A broken instruction: a rule Heal can make clearer, or move into a step or a Skill
skill-useTrajectoryDecision modelHow the Skills the agent fetched worked: whether they were relevant, whether the agent followed them, and whether they helped or misled it. Only runs that fetched a Skill are scoredThree ratings: relevance, whether followed, and impact (helped, no_effect or misled)An irrelevant Skill: its description. An ignored one: the agent's instructions or the Skill's clarity. A misleading one: the Skill's content
over-budgetTrajectoryCodeThe share of turns that cost more than the target in the agent's limits.cost_usd, subagents included. A turn that reports finished work is judged on the total its budget showed: the turns before it, plus the work's cost. Agents with no cost target are not scoredover_budget_rate, with each turn's cost against its targetThe agent's cost target in limits.cost_usd, and what drives each over-target turn
user-follow-upOutcomeDecision modelThe most serious thing the user's follow-up messages said about the agent's answersno_follow_up, accepted, found_incomplete or correctedA correction becomes a memory. An incomplete answer changes what the agent's answers must cover

To score with a builtin by hand, pin a version, or customize one, see Using builtin eval rubrics.

Builtin eval rubrics for builtin agents​

Autoheal's builtin agents come with eval rubrics for what a good run of that agent produces. Each is in an eval file that the agent's agent.yaml attaches:

YAML
evals:
- evals:regression-attribution

They don't only check correctness. Others check safety, restraint, coverage and whether the agent verified its own work. Each agent has several; this table shows one eval file for each of ten common agents:

AgentEval fileWhat it scores
Incident responsemitigation-safetyA proposed mitigation is bounded to one service and revision, names who must approve it, and states the metric that would confirm recovery
Alert triagenotification-restraintThe run told the team that had to act, once, and nobody else: nobody for noise, one note to the owner team after a runbook
Deploy verificationregression-attributionBefore a movement is called the deploy's, the report rules out a schedule, a dependency and a traffic change, and cites the evidence for each
CI/CD failure analyzerpinged-right-personThe person pinged is the one who had to act: the author of the change, the fallback for a platform failure, or no one for a flaky test
Code reviewreview-signal-to-noiseThe review's comments are ones the author would act on: a file, a line and a reason, nits labelled and capped, nothing that restates the diff
Feature developmentfirst-push-greenThe check runs on the first commit the agent pushed all passed. A red check the agent's own change caused is a miss, whatever its sandbox tests said
Vulnerability remediationremediation-minimalityThe proposed change is the smallest one that closes the reachable path, not the newest version, and not a change to services that were never exposed
Release readinessgate-coverageEach gate the agent declares is answered from evidence gathered in this run, or reported as unanswered
Runbook executionrecovery-verificationBefore resolving, the run verified recovery on the service's SLO signal over the full window, and escalated after one failed attempt
Cloud cost optimizationdata-coverage-honestyFigures are bounded to the window the cost data actually covers, and nothing is extrapolated across a gap

Eval rubrics you define specific to your team​

Autoheal can't know your team's standards, so these eval rubrics don't ship with it. You define them yourself, in eval files in evals/, and attach them to the agents your team owns. You choose the sampling rate, the model and the other fields, so you have full control. Pick a standard that a reviewer on your team already enforces by hand, and write it as a rubric.

These examples show the shape, for four kinds of team:

TeamAgentExample eval fileWhat it checks
DatabaseCode reviewmigration-table-contextA review of a schema change states the table's row count and write rate, and cites where each came from, before calling the change safe
InfrastructureCloud cost optimizationheadroom-restraintA right-sizing proposal declines a saving that would remove headroom an earlier incident showed was needed, and says what proves it
AI applicationCoding agent spendmodel-change-benchmarkedA change to a model, prompt or effort level is benchmarked on the team's own tasks before it is proposed, and reports pass rate, cost and time for every setup tested
Web servicesFeature developmentretry-safetyA change adds or loosens no retry, timeout, backoff or pool setting that would multiply how long a call holds its caller's capacity

See Writing eval rubrics for the file format and how to attach an eval file to an agent.

Who evaluates​

An eval file's type says what scores its rubrics. You can define a rubric of any type:

TypeScored byGood forSampling
scriptCode. The result is deterministicCounts and rates computed from the run, such as failed tool calls or cost over targetEvery run
decision_judgeA decision model such as Jev, which answers each rubric by choosing a label, and writes no textQuick checks on every turn: was the claim cited, were the instructions followedDefault 0.5
llm_judgeAn LLM, which reads the conversation and writes a reasoned answer for each rubricDeeper questions, and numeric scoresDefault 0.5

A decision rubric is faster and cheaper than an LLM judge rubric, but it is boolean or categorical, never numeric. When a deployment has no decision model set up, the judge model answers a decision eval file instead.

judge and decision are the old names of llm_judge and decision_judge. An eval file that still says type: judge or type: decision is scored under the new name, and the Git registry shows a warning to rename it.

Coming soon: judges will be agents that read the trace. A judge will read only the messages that settle each rubric, and cite them, so a run longer than the judge's context window can still be scored. looks_for will also narrow a judge to the matching turns.

When eval rubrics run​

Automatically​

When a conversation has been quiet for 10 minutes after a successful turn, Autoheal scores it with every attached eval file: each judge or decision at its sampling rate, each script every time. A conversation that keeps going is scored once, not after every turn.

By hand​

Select Score on a run page and pick up to 20 eval files. Sampling doesn't apply, so every eval file you pick scores the run. You can score with an eval file from another branch or an older commit, which is how you test a new rubric on runs that already happened, without re-running the agent.

In agents with several LLM steps​

Each step is its own conversation, scored with the eval files on that step:

YAML
steps:
draft:
instructions: Draft the incident summary.
evals: [evals:evidence-quality]
polish:
instructions: Tighten the summary.
with:
prompt: ${{ steps.draft.output }}
evals: [evals:evidence-quality, ./evals:tone]

A run of this agent gets three scores, one per step and eval file. A step inside a while loop is scored once per iteration: the Scores table lists every iteration, and the Evals page's cards count each run's last iteration. The Evals page groups the scores by step.

Paused, failed and cancelled LLM steps, and subagents' runs, aren't scored automatically. An LLM step that succeeded is scored even when a later step of the run fails.

What eval results are for​

UseApplies toLooks atExample
Score runsAny finished run of the agentThe output, or how the agent got there"Did the root cause cite evidence from a tool result?"
Label runs (coming soon)Any run, finished or notThe input, the output or the trajectory"Is this a db_feature, an app_feature or an infra_fix?"
Score benchmark runsBenchmark casesThe output or trajectory, compared with the case's known answer when it has one"Does the root-cause artifact name resource contention on payments-db?"

Score runs​

A score says how good or bad a run was. Each rubric sets its own scoring: a number between a minimum and a maximum (numeric), true or false (boolean), or one label from a fixed list (categorical). A script eval file reports its own measures instead, such as a failure rate.

Scores from the builtin and team eval rubrics above are what Heal starts from. See Reading scores for where they show up.

Label runs (coming soon)​

Coming soon. Labeling is not available yet.

A labeling rubric assigns one label to each run based on the kind or complexity of its input, its output or its trajectory. For a QA agent, the labels could be db_feature, app_feature and infra_fix. Use separate rubrics for different traits. A labeling rubric uses the same eval file and scoring path as other rubrics, but returns a label instead of a grade. Other examples include:

  • incidents for your highest-tier customers;
  • runs that finished without writing an artifact;
  • runs where a person corrected the agent;
  • investigations of database incidents versus deploy incidents.

Labels make aggregate scores useful. An overall root-cause score of 0.82 can hide 0.95 on deploy incidents and 0.4 on database incidents. Heal looks for exactly that kind of split.

Score benchmark runs​

A benchmark rubric compares a run with a reference answer that a person approved, such as the root cause the team confirmed for a past incident. It runs only on benchmark cases. A case with no reference answer is still scored: the rubric judges the run on its own terms. A rule can be as simple as "the root-cause artifact names payments-db and resource contention", and the result is pass or fail.

Reading scores​

What a result records​

Each result records the agent version, the eval version, the branch and the run it scored. Scores from two versions of a rubric are never mixed in the same chart.

The Evals page summarizes scores by agent version, eval version, rubric, branch and day.

Querying and comparing scores​

Self-improvement agents will be able to query eval scores with SQL, including comparisons between two sets of runs. Heal uses this to find poor scores and the groups of runs where they cluster.

A comparison page will put the scores of any two sets of runs side by side:

  • runs in a time window, or a list of run ids, on any branch;
  • the runs of a benchmark execution on any branch.

Writing eval rubrics​

The eval file​

This eval file holds one judge rubric and scores an alert-triage agent:

YAML
name: Root cause grouping
description: >-
Alerts that share a cause are one group, linked through a cited dependency edge;
a service taking retries is linked but not blamed.
type: llm_judge
sampling:
rate: 1.0
rubrics:
- name: grouped
prompt: >-
What fraction of the co-firing alerts are placed correctly, in the cause's group
with the edge that links them cited, or in a group of their own when nothing links
them?
scoring:
type: numeric
min: 0
max: 1
FieldWhat it does
typellm_judge, decision_judge or script. See Who evaluates.
rubricsThe eval rubrics in the file. Each has a name, a prompt and its own scoring. A decision rubric is boolean or categorical with 2 to 255 labels, not numeric.
looks_forThe tools and bash commands the eval file is about, such as {tools: [restart_pod]}. A run with none of them is marked not applicable, and no model is asked.
applies_toScore only runs that saved this kind of artifact, such as hypothesis. Other runs are marked not applicable.
modelLLM judge eval files only: the judge model, written the same way as an agent's model, such as claude-sonnet-5. If your deployment doesn't offer that model, the push shows an error that lists the models it does offer. Without it, the environment's default judge model is used. A decision eval file can't set it.
samplingFor LLM judge and decision eval files: what fraction of runs to score automatically. Default is 0.5. Scoring by hand ignores it, and a script eval file scores every run whatever sampling says.

Where eval files live​

PathWho can use itReference
evals/<name>.yamlAny agent in the projectevals:<name>
agents/<agent>/evals/<name>.yamlOnly that agent./evals:<name>
Published by AutohealAny agent, without copyingbuiltin/evals:<name>

An eval file in the agent's folder is part of the agent: editing it creates a new agent version too. Editing a project eval file creates only a new eval version.

Attaching eval files to an agent​

An agent attaches eval files in its agent.yaml:

YAML
evals:
- evals:evidence-quality
- {eval: ./evals:asked-before-restart, auto: false}

A plain reference scores automatically. auto: false keeps the eval file for manual scoring only. It is the way to stop a script eval file from scoring automatically, because scripts ignore sampling.

Using builtin eval rubrics​

Scoring by hand. To score a run with a builtin eval file yourself, attach the published eval file to the agent without copying it:

YAML
evals:
- builtin/evals:user-follow-up

For manual-only scoring, use {eval: builtin/evals:user-follow-up, auto: false} instead of the string attachment. Automatic scoring defaults to on.

Versions. Builtin eval files use the current deployed content for every new manual, automatic, reuse or benchmark request. A result records the builtin address and the content digest, with no eval commit, so branch and commit choices don't apply. Retries of a prepared request keep that request's definition. Project eval files keep their existing commit and version behavior.

Inheritance and customizing. Agents that extend a builtin inherit its private eval attachments. Private builtin eval files cannot be referenced directly. To customize a published eval file, copy its YAML into your project's evals/ folder and attach the project copy.

Changing eval rubrics​

Eval rubrics are files, so you change them like code: on a branch, reviewed in a pull request. Heal proposes rubric changes the same way, when it finds a behavior no rubric watches.

To check that a new rubric measures what you meant, score a few past runs with it from its branch before you merge. Pick some runs that should pass and some that should fail.

Limits​

  • Only single-LLM agents can be scored by hand. Score on a run page and benchmarks score one conversation. An agent with several LLM steps is scored automatically, per step.
  • A loop inside a loop keeps only the last outer pass. An LLM step's iterations are numbered within its innermost loop, so each pass of the outer loop reuses them. Conversations from earlier outer passes aren't scored, and the step's results keep only the last outer pass.
  • A judge is a model. Its verdicts can be wrong. Read the reasoning on a few results before you rely on a new rubric.
  • Observe: the trace a judge reads
  • Heal: what happens after weak or unusual scores
  • Benchmark: where benchmark rubrics run
  • Capabilities: which agents can run and read evals