Skip to main content

Benchmark

A benchmark answers one question: is this change to the agent better? You pick past runs that represent the work, run the changed agent on the same inputs, and score the new runs with the agent's evals.

Eval rubrics on live runs tell you how the agent is doing. They can't tell you how a change would do, because live runs only ever use one version of the agent. A benchmark re-runs the same cases on another version, so you can see what the change did.

note

Benchmarks are in preview. Ask Autoheal support to turn them on for your tenant. Read Limits before you benchmark an agent that writes to other systems.

An example​

You suspect your on-call agent would find root causes more often with a stronger model.

  1. Build a dataset

    Open a past run of the agent and select Add to dataset. Do this for 20 investigations that represent its work, some it got right and some it got wrong, and put them in a dataset called oncall-core.

  2. Change the agent on a branch

    Open the agent in the editor, create the branch try-bigger-model, and change model: in agent.yaml. Commit.

  3. Create a benchmark

    On the Benchmarks page, create a benchmark of the on-call agent on oncall-core. Set its branch to try-bigger-model and its attempts per case to 2.

  4. Run it

    Select Run. Each case's input goes to the agent again, twice. Each run that succeeds is scored with the evals listed in agent.yaml on try-bigger-model. You can watch each run's transcript and score as it comes in, and cancel at any point.

  5. Run it on main too

    Run the same benchmark again with the branch changed to main for this run. Now you have the current agent and the changed one, scored on the same cases.

Datasets and cases​

A dataset is a named set of cases. It doesn't decide how runs are scored: the agent's own evals: list does.

A case is an input to give the agent: the arguments a past run was started with, or an input you write by hand.

  • From a run. Add to dataset on a run page copies the run's input into a new case. It copies only the input that started the run, not later follow-up messages, so pick runs whose first message stands on its own, or edit the case until it does.
  • By hand. Write the input yourself in the case editor.
  • Versions. Editing a case makes a new version. Old versions never change, so an earlier execution still shows exactly what it ran.
  • Archiving. When the data a case depends on is gone, such as logs past their retention period, archive the case. Archiving is permanent: an archived case is never run or edited again. To run the same conversation later, make a new case from a newer run.

Expected answers​

A case will also be able to hold the answer the agent should produce, such as the root cause the team confirmed. In a dataset, either every case has an expected answer or none does. With them, benchmark evals can grade a run against the answer. Today every eval judges the run on its own terms, without a reference answer.

Benchmarks and executions​

A benchmark is one agent on one dataset, plus the settings each run starts from:

SettingWhat it does
Branch or commitWhich version of the agent to run, and whose evals: list to score with
Attempts per caseHow many times to run each case, to see how much results vary
Attempts at onceHow many runs go at the same time

The agent and the dataset never change once the benchmark exists. The settings can be edited.

An execution is one click on Run. You can run every case or pick some, and change any setting for this execution only. Each run of one case is a case attempt.

What an execution pins​

Before the first attempt starts, an execution records:

  • the settings it ran with;
  • the commit its branch pointed to at that moment;
  • the version of each case.

Each attempt's scoring also records the version of every eval it used. Later edits to the benchmark, the cases or the evals don't change a finished execution, so two executions stay comparable.

An execution pins code and inputs, not the world. Memories, your integrations' data and Autoheal itself can all change between two executions.

Scoring​

Each attempt whose run succeeds is scored once, with every eval in the agent's evals: list at the execution's commit:

  • Every listed eval is used. Sampling rates and auto: false are ignored, so every run gets every score.
  • Each branch uses its own evals. You can test a change to an eval by changing it on a branch and running the benchmark there.
  • Failures are kept apart. A run that failed, timed out or was cancelled has no score; its run status is the result. A run that succeeded but whose scoring failed still counts as succeeded, with the scoring failure shown in its own column.

There's no rescoring: each attempt is scored once.

The agent must list at least one eval, or Run is refused, because nothing could score its runs.

Comparing executions​

You'll be able to put two executions side by side. Results are matched on the same case version, the same eval version and the same score field. Anything that doesn't match on both sides is shown apart, not dropped:

  • cases that ran on only one side;
  • evals that one branch changed, so their versions differ;
  • failed, unscored and not-applicable results, which are never counted as zero.

Until then, open each execution and compare their scores by eye.

Keeping benchmark runs apart from production​

Benchmark runs are real agent runs. They call your integrations, write artifacts and record tool calls. Benchmark runs will:

  • not generate or change memories, or change how often a memory counts as used;
  • stay out of related-session lookups that normal runs make;
  • not be scored by automatic evals, and stay off the agent's Evals screens;
  • be blocked from writing to outside systems, through tools or the shell;
  • run on their own workers, so a large execution doesn't slow down production runs.

Benchmarking a commit​

When you commit a change to an agent, Autoheal will suggest benchmarking it. If you committed to main, it'll create a branch for the benchmark; otherwise it'll use your branch. You'll pick the dataset, and the branch is filled in for you.

Heal will use benchmarks the same way, running one execution per candidate fix and comparing them.

Limits​

  • Benchmark runs aren't isolated yet. Until isolation ships, a benchmark run behaves like any other run: it can propose memories, show up in related sessions, and get scored a second time by automatic evals. Its tools can write to your systems. Benchmark only agents whose tools are read-only.
  • Cases replay against live data. Tools query your integrations as they are now, not as they were when the original run happened. A log line that has since rotated out, or a dashboard that changed, changes what the agent sees.
  • The agent can take shortcuts. A memory or a past investigation may already hold the answer to a case. A higher score can mean the agent remembered rather than worked it out.
  • One LLM step only. Only agents with a single LLM step can be scored, so only those can be benchmarked.
  • 200 attempts per execution. Cases × attempts per case can't exceed 200.
  • One input per case. A case can't replay a conversation with several follow-up messages.
  • No budgets. A case can't be stopped for costing too much.

Coming later​

  • Offline replay. Tool results recorded from the original run and played back, so every execution sees exactly the same data.
  • Pinned memory. A case sees only memories and past runs from before its source run.
  • More ways to make cases, beyond copying one run at a time, including cases built from several turns.
  • Evaluate: the evals a benchmark scores with
  • Heal: uses benchmarks to test the fixes it proposes
  • Capabilities: which agents can run benchmarks and write datasets