Skip to main content

2. Write an Eval

To know if a change made the agent better, you need a test that gives the same check each time. An eval is that test. It is a YAML file with questions about a run, called rubrics. A model, called the judge, reads the run and answers each question. In this page, you write an eval for the example pull request, score runs with it, change the instructions, and compare the scores.

The example pull request has four problems that we know about:

  • The timeout value is in milliseconds, but the gRPC call takes seconds.
  • The README names the setting CATALOG_TIMEOUT_MS, but the code reads RECOMMENDATION_CATALOG_TIMEOUT_MS.
  • A catch-all except Exception hides errors from the product catalog.
  • The pull request adds no test.

A good review finds all four. The eval checks for each one.

Step 1: Create the eval​

  1. Open the New eval dialog

    Go to Agent Studio → Editor. Click New at the bottom of the file tree, and select New eval.

  2. Name the eval

    In Eval ID, type catches-known-bugs and click Create. The editor opens evals/catches-known-bugs.yaml with an LLM judge template.

  3. Write the rubrics

    Replace the content of the file with this eval:

YAML
name: Catches known bugs
description: The review of the example pull request finds the four problems that we know it has.
type: llm_judge
sampling:
rate: 1
rubrics:
- name: found_unit_bug
prompt: >-
Does the review say that the timeout value is in milliseconds, but the
gRPC call takes seconds?
scoring:
type: boolean
- name: found_setting_name
prompt: >-
Does the review say that the setting name in the README is different
from the name that the code reads?
scoring:
type: boolean
- name: found_hidden_errors
prompt: >-
Does the review say that the catch-all exception handler hides catalog
errors?
scoring:
type: boolean
- name: found_missing_test
prompt: Does the review say that the pull request adds no test?
scoring:
type: boolean
- name: verdict
prompt: What verdict does the review give?
scoring:
type: categorical
labels: [merge, merge_with_changes, do_not_merge]

Commit the file.

FieldWhat it does
typellm_judge asks an LLM each rubric's question, and it writes a reason for each answer. decision_judge asks a decision model instead: it is faster and cheaper, writes no reason, and answers only yes/no and label questions. See Who evaluates.
sampling.rateThe share of runs that Autoheal scores automatically, from 0 to 1. 1 scores every run. Without sampling, Autoheal scores half of the runs.
rubrics[].nameThe key of the score. Use letters, digits and underscores. Each name must be unique in the file.
rubrics[].promptThe question that the judge answers about the run.
rubrics[].scoring.typeboolean (true or false), numeric (a number from 0 to 1) or categorical (one of labels).

Step 2: Attach the eval to the agent​

In agents/pr-review/agent.yaml, add this field at the end:

YAML
evals:
- evals:catches-known-bugs

evals:<ID> points to evals/<ID>.yaml at the root of the project. Commit the change.

Step 3: Run the agent and score the run​

Run the agent with the example pull request URL. When the run ends, you can score it in two ways:

  • By hand. In the run's panel, open the Evals card and click Score run. In the Score this run dialog, select the branch, select Catches known bugs, and click Score. The results show in the card in less than a minute.
  • Automatically. Autoheal scores a run with each attached eval when the run's conversation is quiet for 10 minutes. With rate: 1, it scores every run. Automatic scores use the eval files on the default branch, usually main.
An eval on a branch gets no automatic scores

Automatic scoring reads the eval file from the default branch. If the eval file is only on your branch, the automatic attempt ends with "no eval could be used". Score by hand and select your branch in the Score this run dialog.

A score by hand ignores sampling. You can score a run with an eval that the agent does not attach, or with the eval at another branch or commit. Use this to try a new rubric on old runs before you attach it.

Step 4: Read the result​

The Evals card shows one result for each rubric, and the judge's reason for it. A result of the review in this page looked like this:

RubricScoreJudge's reason (short)
found_unit_bugtrueThe review says that timeout= takes seconds, and that 2000 becomes about 33 minutes.
found_setting_nametrueThe review names both setting names.
found_hidden_errorstrueThe review says that except Exception hides every catalog error.
found_missing_testtrueThe review says that the diff adds no test.
verdictmerge_with_changesThe review says "merge with changes".

Read the reason for each score, not only the score. A judge is a model, and it can be wrong. If a reason does not match the review, make the rubric's question more exact.

Click View all evals to open the agent's Evals tab. It has one card for each rubric, with the scores over time. The Scores table below the cards has one row for each run and rubric, with the Score and the judge's Reasoning.

Step 5: Change the instructions and compare​

A smaller review can be faster and cheaper. Test if it is also as good. In instructions.md, replace step 3:

MARKDOWN
3. Read only the diff. Do not open other files.

Commit, and run the agent two or three times with the same URL. Score each run, or wait for the automatic scores. Each commit that changes the agent makes a new agent version.

On the Evals tab, click the card of a rubric, for example found_unit_bug. The By agent version panel shows the scores of each agent version next to each other. On the charts, dashed lines mark new agent versions. Compare the two versions:

  • Check that each version found each of the four bugs.
  • Check if the verdict changed. In our runs, one run of the smaller review gave do_not_merge, and all runs of the full review gave merge_with_changes.
  • Compare Cost per run in Insights, or the Limits card of each run. See page 1.

One run is not enough to decide. Models give different answers to the same input. Run each version a few times, and compare the share of runs that pass. Keep the change only if the scores do not drop.

How the judge works​

  • The judge reads the run's conversation, with the tool calls, and scores the agent's latest reply. It answers each rubric with a short reason first, and then the score.
  • The judge model is the default judge model of your Autoheal installation. To use another model, add model: to the eval file, for example model: claude-sonnet-5.
  • Write a boolean rubric so that true is the good answer. The Evals tab shows true in green and false in red.
  • A numeric score is always from 0 to 1.
  • Each score records the agent version and the eval version. When you change a rubric, its new scores do not mix with the old ones.
  • The eval type decision_judge asks a faster model the same questions. It can give only boolean and categorical scores.

What to know​

  • An eval in agents/<agent>/evals/<ID>.yaml belongs only to that agent. Attach it with ./evals:<ID>.
  • To score an eval only by hand, attach it as {eval: evals:catches-known-bugs, auto: false}.
  • You can score by hand only an agent that has one model step. An agent with steps gets automatic scores for each model step.
  • Autoheal also has eval files for every agent, for example instruction-use and skill-use. See Builtin eval rubrics for all agents.
  • Benchmarks and Datasets run a changed agent again on a set of past runs. They are off by default, and your admin can turn them on. See Benchmark.

From the CLI​

The CLI has no evals command. Push the agent with the CLI, and score a run with autoheal api:

BASH
autoheal agents push agents/pr-review -m "Read only the diff"
autoheal run pr-review "https://github.com/autohealai/astronomy-shop-template/pull/1" --wait -o json
autoheal api POST /api/v1/evals/sessions/<session ID>/score --project <project ID> \
--field 'evals=["evals:catches-known-bugs"]' --field branch=<branch>
autoheal api GET /api/v1/evals/requests/<request ID> --project <project ID>

The run's JSON has the session_id. The score request's JSON has the id of the request. When the request is done, its evals[].scores have each rubric's value and the judge's rationale.

See Evaluate for all eval fields, and Evals in the agent reference.