Skip to main content

Self-Improvement

An agent that worked last quarter can be wrong today. The services it investigates get new dependencies, an API changes shape, a team renames its dashboards. Self-improvement is how Autoheal notices that and fixes the agent's context before people stop trusting it.

It works in five steps. Each one has its own page.

StepWhat happensScope
ObserveEvery run gets a trace: totals, a headline per turn, and SQL access to every message and tool callOne run
EvaluateRubrics from your Git repository score runs or identify run features for analysisOne run at a time, summarized across many
LearnA finished run is reflected on, and turned into memories or a skillOne run, or a few you pick
BenchmarkA proposed change is re-run on past cases and compared with the current agentA dataset of cases
HealWeak or unusual scores across many runs are diagnosed and fixed with a pull requestMany runs
Every run
Observe
Trace each run
Evaluate
Score runs with rubrics
Tactical · one run
Learn
Reflect on a finished run
proposes
Memories or a skill
review queuepull request
Strategic · many runs
Heal
Diagnose weak scores
proposes a fix
Benchmark
Re-run past cases on the fix
scores improve
Pull request
agent.yamlskilltemplaterubricbenchmark
an engineer approves
Next runs
Use the new memory, skill or agent version
Next runs are traced and scored

Tactical and strategic​

There are two speeds of improvement, and they answer different questions.

Learning is tactical. It looks at one run, or a handful, right after they finish, and asks what's worth keeping. A person corrected the agent; the agent spent 40 tool calls finding which cluster a service runs on; a procedure worked and could be repeated. That becomes a memory or a skill that the next run can use.

Healing is strategic. It looks at hundreds of runs through their eval scores and asks what's wrong with the agent itself. Root-cause accuracy dropped after a model change; runs on database incidents cost three times the average; no run on a tier-1 customer's incident produced a postmortem. The fix is a change to the agent: its agent.yaml, a skill, an artifact template, or the rubrics and benchmarks that watch it.

A memory is cheap to add and easy to throw away. A change to the agent affects every future run. So learning runs on its own after each run, and healing goes through a benchmark and a reviewer.

Everything is in Git​

Agents, skills, eval rubrics and artifact templates are files in your project's Git repository. That has three consequences:

  • A change is a branch. Healing proposes a fix on a branch. You review it like any other pull request.
  • Scores carry versions. Every eval result records which agent version and which rubric version produced it, so a score from before a change is never mixed with one from after.
  • One write path. The agents that improve other agents don't need a separate tool for each kind of file. They edit files in a branch. See Capabilities.

Memories are the exception. They live in the memory store and go through the Memories review queue.

Engineers approve every change​

Nothing changes an agent's behavior without a person approving it:

  • A skill, agent.yaml, artifact template, rubric or benchmark change is a pull request.
  • A proposed memory waits in the Memories review queue, under that page's auto-accept rules.

Agent tool calls that write to your systems, like a rollback, are governed separately by approval policies.

Limits​

The loop only improves what a rubric can see. If no rubric checks for a failure, and no person corrects it in a run, the failure stays invisible. Writing a rubric for it is usually the first fix. See Evaluate.