Build Your Own Agents
An agent you build automates a repeatable post-coding task that no pre-built agent covers. You give Autoheal a trigger, a goal, a budget and the tools an agent may reach. The agent works that task and ships something an engineer reviews. Every run is scored.
You own every part of the agent: its behavior, tools, models and planning. Evals calibrate its performance over time, so you can see whether a change, from a model swap to splitting work into subagents, actually helped. Recurring patterns become new skills, and every change requires your approval.
Each agent is an agent.yaml file, with the files it refers to, in your project's Git repository. You create and edit agents in Agent Studio → Agents or with the CLI. Runs lists every run.
Define an agent
An agent is defined by four parts. Together they set when it runs, what it is trying to achieve, what it may spend, and what it may touch.
| Part | What you set |
|---|---|
| Trigger | A GitHub event, a ServiceNow incident or a new alert group |
| Goal | The outcome you want, written as a prompt |
| Budget | Default model, reasoning ceiling and spend per run |
| Tools | The Autoheal and integration tools it may call |
Trigger
A trigger decides when the agent runs, and an agent runs whenever any of its triggers fire. You can trigger on:
- A GitHub event from a connected GitHub integration, such as a pull request merged or a release published on a specific repository.
- A ServiceNow incident created or changed in a connected ServiceNow instance.
- A new alert group from any connected alert source, filtered on fields such as its severity.
Triggers are managed on the Triggers page or with the CLI. You can also start a run by hand from the agent's page or with the CLI.
Goal
The goal is the outcome you want, written as a prompt. In the agent builder it is the agent's Instructions. A regression watcher's goal might read: "After each merge to main, find any production regression the change caused. Compare error rate, p99 latency and saturation for the affected services against the deploy timestamp. Check Sentry for new error signatures. Report the likely cause with a recommended rollback or fix."
The goal is stable within a run, but it isn't frozen. Eval rubrics score its runs. When scores drop, Heal proposes a change to the agent and tests it on a benchmark of your own past runs. No change goes live until an engineer approves it, so the agent improves over time without changing under you unannounced.
Budget
The budget sets what a run may cost:
- Default model. Each agent picks its model from your approved list. Subagents can use a separate default. This lets you keep the frontier model for the reasoning that needs it, and route routine steps to a cheaper one.
- Reasoning ceiling. How much thinking the model may do per step.
- Spend per run. A cap on what a single run may spend, alongside a budget per agent.
Tools
Tools are the calls the agent may make. Two fields in the agent's agent.yaml choose them, and the platform, not the prompt, decides which tools the agent is given:
system_toolschooses among Autoheal's own tools, such asget_incident,create_artifactorgit_commit.integration_toolschooses among the tools of your connected integrations, such asdatadog_search_logs.
Each field is a filter: one expression that is checked once for each tool. The agent keeps a tool when the expression is true for it. A filter can test the tool's name and its operation_type (read or write). integration_tools can also test its integration_type:
# Autoheal's read tools, plus saving findings as an artifact
system_tools: operation_type == "read" || name == "create_artifact"
# Every integration's read tools, plus GitHub writes, such as creating an issue
integration_tools: operation_type == "read" || integration_type == "github"
| Each field | What the agent can call |
|---|---|
| Left out | Every tool of the group, reads and writes, including tools Autoheal adds later |
"false" | No tool of the group |
operation_type == "read" | Read tools only |
An agent with neither field can call every tool. That is every system tool, write tools such as git_commit, update_incident, delete_artifact and create_oncall_page (which pages people) included, and every tool of every connected integration. For an agent that only looks and reports, write system_tools: operation_type == "read". An agent that an alert or a GitHub event starts may have nobody to answer a question, and a question pauses the run until someone answers it. For such an agent, wrap system_tools in parentheses and add && name != "ask_user_input": without the parentheses, a filter with || still keeps the tool. An agent that a Slack message starts can keep it: its questions, and the approvals its calls wait for, are posted in that Slack thread with buttons to answer them.
integration_tools chooses among the integrations the run can see: every active integration of its project, such as GitHub, Datadog, Grafana or LaunchDarkly. The credential you connect sets the ceiling. Many integrations ship only read tools, so for those a write filter adds nothing. Your organization's settings decide which system tools exist. For example, the incident tools exist only when Incidents is enabled for your organization.
The agent is shown only the tools its filters keep, and the platform refuses a call to any other tool. A subagent is also limited by the filters of the agent that calls it, for the tools its model chooses. A tool step written in a subagent's steps is not limited by the caller's filters.
Tool filters do not limit the shell. Every agent can run shell commands in its sandbox, and the shell uses the project's CLI connections. A filter such as integration_tools: operation_type == "read" does not stop a shell command from writing to a system that one of those connections reaches. Governance policies can require a person's approval before a matching integration tool call or shell command runs, such as every write.
The reference has the filter language, the checks on push, and the filter to write for each rule of the removed tools field.
Example jobs
These are the kind of jobs an agent you build takes off a Platform Engineering, DevEx, DevOps or SRE team.
| Job | What the agent does |
|---|---|
| Stale feature flag cleanup | Finds flags with no evaluations in 60 days and opens one PR per flag with the dead branch removed and tests updated |
| Deprecated internal API | An API is sunset, but its callers sit in forty repos nobody has time to touch. The agent finds every caller and opens one migration PR per repo, with tests run |
| CODEOWNERS and on-call drift | Stale CODEOWNERS stall reviews and page departed teammates, so on offboarding the agent finds every entry, suggests successors from commit history, and opens a PR |
| Flaky test quarantine | When CI passes on retry, quarantines the flaky test with logs attached, and opens a fix PR once the cause reproduces |
If a task can be written as a goal and grounded in your engineering context, an agent can run it.
What it does once running
| What happens | Where it's covered | |
|---|---|---|
| Output | Each run produces something a person can review, reject or merge. This is a pull request with tests run, or a finding with its evidence and confidence score | Runs and results |
| Scoring | Each run is scored against your evals, and weak scores lead to proposed context changes that an engineer approves | Evaluator, Healer |
| Shared context | Memories, skills and learned integration paths are available to other agents, so a second agent for a similar job starts with that context | Memories, Skills |
| Controls | The same controls as every other agent: tool filters, approval policies, and a budget per run and per agent | Governance Policies |
Evaluation
Evals give each change to an agent a measurable result. A model swap or a restructuring into subagents is then judged on scores, not impressions.
| Feature | What it does |
|---|---|
| Built-in evals | Score every run's trajectory and outcome without any setup |
| Custom judges | Rubric-based judges you write and attach to an agent, scored as a number, a pass/fail or a category |
| Score history | Daily trends per eval, with comparison across agent versions |
| Proposed skills | Recurring patterns, such as a retry storm, can become skills, which go live only after an engineer approves them |
Evals only measure what your rubrics and downstream signals capture. A failure mode that nobody wrote a rubric for stays invisible until someone notices it in a run.
Where it runs
An agent you build runs on the same platform as the Incident Response, Vulnerability Remediation and AI Coding Cost Efficiency agents. It runs in your isolated tenant (SaaS) or your own environment (BYOC), on models from your approved list. See Sovereignty for SaaS and BYOC, connected or airgapped.
- Built-in harness. Agents run on Autoheal's built-in harness, with each run's work isolated in a sandbox. Support for running agents on the coding agents your teams already use is in development (coming soon).
- Bring your own models. Route each task to the right model from your approved list, with per-agent budgets, a separate default for subagents, and an allowlist the platform enforces.
- Start it from where the work happens. A GitHub event, a ServiceNow incident or a new alert group starts it through a trigger. Anyone with access can also run it from its page or the CLI.
Runs and results
Every run is recorded under Runs. A run shows:
- Its status: completed, waiting for input, failed or stopped. After you send a follow-up message to a run, the run shows the status of that message.
- The trigger that started it.
- The tools it used.
- Its duration.
- A result summary, such as "Opened 12 PRs for stale flags" or "3 new flaky tests in the checkout suite".
Next to the filters, four counts show the runs that are running now, the runs that wait for input, and the runs that failed or succeeded in the last 7 days. Click a count to show only the runs that it counts. Click it again to show all runs. The search, the filters and the page are in the URL, so a link shows the same runs to the person who opens it.
Opening a run shows the full trace: each step the agent took, the context it loaded, the tools it called, and the conclusion it reached.
When a run needs a decision, it pauses as waiting for input instead of acting unattended. Agents post their outcomes where the work already happens. Examples are a pull request, a PR comment, or a message in a Slack or Microsoft Teams channel.
Get started
- Create an agent in Agent Studio → Agents, or with
autoheal agents createin the CLI. - Connect the integrations the agent needs.
- Write the goal.
- Set the budget: default model, reasoning ceiling and spend per run.
- Write the
system_toolsandintegration_toolsfilters for what the agent may call. - Attach evals, then try the agent and publish it.
- Add a trigger on a GitHub event, a ServiceNow incident or a new alert group.