Skip to main content

Self-Improving Agent Spec

An agent spec is the agent's definition: its instructions, the tools it may use, its reasoning effort, and its model routing. It's the how of an agent, as distinct from the context it reasons over.

Specs decay if left alone. Your stack moves, an integration changes shape, a service is renamed, and a prompt that scored well last quarter quietly starts producing worse conclusions. Autoheal treats the spec as a production artifact with continuous measurement and a review gate, so drift surfaces as a concrete proposal instead of a slow decline.

The loop is Run Private Evals → Update → Test & Rollout, and it closes on itself: the rollout of one update is measured by the same evals that triggered it.

1. Run private evals​

Every agent run is scored inside your own environment, rather than a sample of runs or a nightly batch over some of them.

Each run is graded on two axes:

AxisQuestion it answersWhat it catches
TrajectoryHow did the agent work?Wrong data sources, wasted or redundant steps, an unsound diagnostic path, tools queried in a senseless order
ArtifactWhat did the agent produce?A root cause that isn't evidence-backed, an incomplete RCA, a fix proposal that doesn't address the defect

Both matter, and they fail independently. An agent can reach the right conclusion by luck through a terrible trajectory, and that will keep working until it doesn't. It can also follow a textbook path and still write up a conclusion the evidence doesn't support. Scoring both is what makes the signal trustworthy.

Scores are tracked per agent over time, so you're looking at a trend rather than a single verdict.

note

Evals are private to your tenant. They run in your environment against your runs, and your run data is never used to train models.

2. Update​

A sustained score drop is a regression, and a regression proposes an update to the agent definition.

This is the part that distinguishes a measured agent from a maintained one. Nobody has to notice that quality slipped, form a theory about which instruction went stale, and hand-edit a prompt. The eval scores localize the problem, and Autoheal proposes the change as a reviewable diff against the current spec, with the failing scores that motivated it.

An update proposal is never applied on its own. It goes to the next stage first.

3. Test & rollout​

Before a proposed spec goes live, it is validated two ways.

Replay against past runs​

The candidate spec is replayed against a golden dataset: good and bad past runs selected from your own environment.

1
Curate

You assemble a golden dataset from real past runs, both the ones the agent got right and the ones it got wrong. Both matter: the failures are what prove an update fixed something.

2
Replay

The candidate spec is re-run against that dataset and scored with the same trajectory and artifact rubrics as live runs.

3
Compare

You see a before-and-after against the current spec on cases you recognize, rather than an abstract claim of improvement.

Replay is fast and deterministic, but it only tells you how the candidate handles situations that already happened.

Run in shadow​

So the candidate also runs in shadow: it executes alongside the live agent on real incoming work, producing its own conclusions without acting on them or reaching your team. This is what catches problems a golden dataset can't, namely the candidate's behavior on cases nobody thought to curate.

The rollout gate​

Only after replay and shadow does an engineer approve rollout. That approval is required, since a benchmarked, shadow-tested spec still doesn't promote itself.

Approval is scoped at the team level, so each team governs the specs its own agents run under. See Sovereignty & Deployment for how approval policies are configured.

Agents as code​

The upshot of versioning specs, reviewing diffs, and benchmarking changes is that managing an agent looks like managing a service. Your platform engineers apply review discipline they already have, reading the diff, checking the benchmark, and approving the rollout, so they become AI-skilled through work they already know how to do rather than by learning prompt engineering as a separate craft.