Skip to main content

Heal

Healing is the strategic part of self-improvement. Learn adds memories and skills from a single run. Heal looks at hundreds of runs and changes the agent itself.

It works in three steps:

  1. Reflect on eval scores to find what's poor or what changed.
  2. Diagnose and fix: find the cause, and propose a change on a branch.
  3. Validate the change before anyone reviews it.

Each step is done by an agent defined in agent.yaml, like any other Autoheal agent. You can read its instructions, change them, and attach evals to it.

Reflect​

The healer reads eval scores for an agent overall and across groups of runs, and asks:

  • Which scores are poor? Rubrics where the agent is consistently weak.
  • What changed? A score that dropped after a new agent version, a model change, or a change in the services it works on.
  • Where does it differ? A group of runs that scores much worse than the rest. Labeling evals label traits such as tier-1 customers or database incidents.
  • What costs too much? Runs that use far more tool calls or money than others of the same kind.

For example: root-cause accuracy is 0.91 overall, but 0.42 on incidents in the payments namespace since agent version 14.

Diagnose and fix​

For each problem, the healer opens a sample of the affected runs through their traces and compares them with runs that went well. It writes down a hypothesis, such as "since v14 the agent skips the deploy history when the alert names a database", with the runs and message ids that support it.

Then it proposes a fix. There are three kinds, depending on what's missing.

The problem isFixWhat changes
The agent does the wrong thingChange the agentagent.yaml (instructions, model, tools, subagents), a skill, or an artifact template
Nobody would notice if it didImprove observabilityA new or sharper eval rubric
A change could break it again unnoticedImprove testingNew benchmark cases or evaluators

A single problem often needs more than one. A regression that went unnoticed for two weeks needs the fix, a rubric that would have caught it, and a benchmark case so the next change can't bring it back.

Each fix goes on its own branch, so you can accept one and decline another.

Validate​

Each kind of fix is checked differently before it reaches review.

FixHow it's checked
Agent changeRun the benchmark on the fix's branch, and compare its scores with the current agent's on the same cases. The fix should improve the target score without lowering others.
New or changed rubricScore past runs with the rubric from its branch. It should pass the runs that behaved well and fail the ones that didn't.
New benchmark cases or evaluatorsRun the benchmark with the current agent. The new cases should fail where the agent is known to be weak, and the new evaluators should tell good runs from bad ones.

When the healer tries several fixes for the same problem, it runs one benchmark execution per branch and compares them with each other, and with the last execution on the agent version where the problem was found.

Engineer approval​

Every fix arrives as a pull request with:

  • the scores that showed the problem;
  • the hypothesis, with links to the runs and messages behind it;
  • the validation results.

Nothing merges without an engineer's approval. Because the change lands in shared context, an approved fix to a skill or template helps every agent that reads it.

Running it​

The healer can run:

  • on a schedule, for example nightly over the last day's runs;
  • when a score moves, if a rubric's average drops past a threshold;
  • when you ask, from an agent's Evals page or by chatting with the assistant: "why did root-cause accuracy drop last week?"

Limits​

  • It can only fix what a rubric measures. If nothing scores a behavior, the healer can't see it get worse. That's why rubric changes are one of its three kinds of fix.
  • A benchmark covers only the cases in it. A fix can pass the benchmark and still behave differently on work the dataset doesn't contain. That's one reason a person approves every change.
  • Evaluate: the scores healing starts from
  • Benchmark: how a fix is tested
  • Learn: the faster, single-run counterpart
  • Capabilities: the Git and benchmark capabilities the healer uses