Skip to main content

Build Your Own Agents

An agent you build automates a repeatable post-coding task that no pre-built agent covers. You give Autoheal a trigger, a goal, a budget and the tools an agent may reach. The agent works that task and ships something an engineer reviews. Every run is scored.

You own every part of the agent: its behavior, tools, models and planning. Evals calibrate its performance over time, so you can see whether a change, from a model swap to splitting work into subagents, actually helped. Recurring patterns become new skills, and every change requires your approval.

Each agent is an agent.yaml file, with the files it refers to, in your project's Git repository. You create and edit agents in Agent Studio → Agents or with the CLI. Runs lists every run.

Define an agent​

An agent is defined by four parts. Together they set when it runs, what it is trying to achieve, what it may spend, and what it may touch.

PartWhat you set
TriggerA GitHub event, a ServiceNow incident or a new alert group
GoalThe outcome you want, written as a prompt
BudgetDefault model, reasoning ceiling and spend per run
ToolsThe Autoheal and integration tools it may call

Trigger​

A trigger decides when the agent runs, and an agent runs whenever any of its triggers fire. You can trigger on:

  • A GitHub event from a connected GitHub integration, such as a pull request merged or a release published on a specific repository.
  • A ServiceNow incident created or changed in a connected ServiceNow instance.
  • A new alert group from any connected alert source, filtered on fields such as its severity.

Triggers are managed on the Triggers page or with the CLI. You can also start a run by hand from the agent's page or with the CLI.

Goal​

The goal is the outcome you want, written as a prompt. In the agent builder it is the agent's Instructions. A regression watcher's goal might read: "After each merge to main, find any production regression the change caused. Compare error rate, p99 latency and saturation for the affected services against the deploy timestamp. Check Sentry for new error signatures. Report the likely cause with a recommended rollback or fix."

The goal is stable within a run, but it isn't frozen. Eval rubrics score its runs. When scores drop, Heal proposes a change to the agent and tests it on a benchmark of your own past runs. No change goes live until an engineer approves it, so the agent improves over time without changing under you unannounced.

Budget​

The budget sets what a run may cost:

  • Default model. Each agent picks its model from your approved list. Subagents can use a separate default. This lets you keep the frontier model for the reasoning that needs it, and route routine steps to a cheaper one.
  • Reasoning ceiling. How much thinking the model may do per step.
  • Spend per run. A cap on what a single run may spend, alongside a budget per agent.

Tools​

Tools are the calls the agent may make. Two fields in the agent's agent.yaml choose them, and the platform, not the prompt, decides which tools the agent is given:

  • system_tools chooses among Autoheal's own tools, such as get_incident, create_artifact or git_commit.
  • integration_tools chooses among the tools of your connected integrations, such as datadog_search_logs.

Each field is a filter: one expression that is checked once for each tool. The agent keeps a tool when the expression is true for it. A filter can test the tool's name and its operation_type (read or write). integration_tools can also test its integration_type:

YAML
# Autoheal's read tools, plus saving findings as an artifact
system_tools: operation_type == "read" || name == "create_artifact"

# Every integration's read tools, plus GitHub writes, such as creating an issue
integration_tools: operation_type == "read" || integration_type == "github"
Each fieldWhat the agent can call
Left outEvery tool of the group, reads and writes, including tools Autoheal adds later
"false"No tool of the group
operation_type == "read"Read tools only

An agent with neither field can call every tool. That is every system tool, write tools such as git_commit, update_incident, delete_artifact and create_oncall_page (which pages people) included, and every tool of every connected integration. For an agent that only looks and reports, write system_tools: operation_type == "read". An agent that an alert or a GitHub event starts may have nobody to answer a question, and a question pauses the run until someone answers it. For such an agent, wrap system_tools in parentheses and add && name != "ask_user_input": without the parentheses, a filter with || still keeps the tool. An agent that a Slack message starts can keep it: its questions, and the approvals its calls wait for, are posted in that Slack thread with buttons to answer them.

integration_tools chooses among the integrations the run can see: every active integration of its project, such as GitHub, Datadog, Grafana or LaunchDarkly. The credential you connect sets the ceiling. Many integrations ship only read tools, so for those a write filter adds nothing. Your organization's settings decide which system tools exist. For example, the incident tools exist only when Incidents is enabled for your organization.

The agent is shown only the tools its filters keep, and the platform refuses a call to any other tool. A subagent is also limited by the filters of the agent that calls it, for the tools its model chooses. A tool step written in a subagent's steps is not limited by the caller's filters.

Tool filters do not limit the shell. Every agent can run shell commands in its sandbox, and the shell uses the project's CLI connections. A filter such as integration_tools: operation_type == "read" does not stop a shell command from writing to a system that one of those connections reaches. Governance policies can require a person's approval before a matching integration tool call or shell command runs, such as every write.

The reference has the filter language, the checks on push, and the filter to write for each rule of the removed tools field.

Example jobs​

These are the kind of jobs an agent you build takes off a Platform Engineering, DevEx, DevOps or SRE team.

JobWhat the agent does
Stale feature flag cleanupFinds flags with no evaluations in 60 days and opens one PR per flag with the dead branch removed and tests updated
Deprecated internal APIAn API is sunset, but its callers sit in forty repos nobody has time to touch. The agent finds every caller and opens one migration PR per repo, with tests run
CODEOWNERS and on-call driftStale CODEOWNERS stall reviews and page departed teammates, so on offboarding the agent finds every entry, suggests successors from commit history, and opens a PR
Flaky test quarantineWhen CI passes on retry, quarantines the flaky test with logs attached, and opens a fix PR once the cause reproduces

If a task can be written as a goal and grounded in your engineering context, an agent can run it.

What it does once running​

What happensWhere it's covered
OutputEach run produces something a person can review, reject or merge. This is a pull request with tests run, or a finding with its evidence and confidence scoreRuns and results
ScoringEach run is scored against your evals, and weak scores lead to proposed context changes that an engineer approvesEvaluator, Healer
Shared contextMemories, skills and learned integration paths are available to other agents, so a second agent for a similar job starts with that contextMemories, Skills
ControlsThe same controls as every other agent: tool filters, approval policies, and a budget per run and per agentGovernance Policies

Evaluation​

Evals give each change to an agent a measurable result. A model swap or a restructuring into subagents is then judged on scores, not impressions.

FeatureWhat it does
Built-in evalsScore every run's trajectory and outcome without any setup
Custom judgesRubric-based judges you write and attach to an agent, scored as a number, a pass/fail or a category
Score historyDaily trends per eval, with comparison across agent versions
Proposed skillsRecurring patterns, such as a retry storm, can become skills, which go live only after an engineer approves them

Evals only measure what your rubrics and downstream signals capture. A failure mode that nobody wrote a rubric for stays invisible until someone notices it in a run.

Where it runs​

An agent you build runs on the same platform as the Incident Response, Vulnerability Remediation and AI Coding Cost Efficiency agents. It runs in your isolated tenant (SaaS) or your own environment (BYOC), on models from your approved list. See Sovereignty for SaaS and BYOC, connected or airgapped.

  • Built-in harness. Agents run on Autoheal's built-in harness, with each run's work isolated in a sandbox. Support for running agents on the coding agents your teams already use is in development (coming soon).
  • Bring your own models. Route each task to the right model from your approved list, with per-agent budgets, a separate default for subagents, and an allowlist the platform enforces.
  • Start it from where the work happens. A GitHub event, a ServiceNow incident or a new alert group starts it through a trigger. Anyone with access can also run it from its page or the CLI.

Runs and results​

Every run is recorded under Runs. A run shows:

  • Its status: completed, waiting for input, failed or stopped. After you send a follow-up message to a run, the run shows the status of that message.
  • The trigger that started it.
  • The tools it used.
  • Its duration.
  • A result summary, such as "Opened 12 PRs for stale flags" or "3 new flaky tests in the checkout suite".

Next to the filters, four counts show the runs that are running now, the runs that wait for input, and the runs that failed or succeeded in the last 7 days. Click a count to show only the runs that it counts. Click it again to show all runs. The search, the filters and the page are in the URL, so a link shows the same runs to the person who opens it.

Opening a run shows the full trace: each step the agent took, the context it loaded, the tools it called, and the conclusion it reached.

When a run needs a decision, it pauses as waiting for input instead of acting unattended. Agents post their outcomes where the work already happens. Examples are a pull request, a PR comment, or a message in a Slack or Microsoft Teams channel.

Get started​

  1. Create an agent in Agent Studio → Agents, or with autoheal agents create in the CLI.
  2. Connect the integrations the agent needs.
  3. Write the goal.
  4. Set the budget: default model, reasoning ceiling and spend per run.
  5. Write the system_tools and integration_tools filters for what the agent may call.
  6. Attach evals, then try the agent and publish it.
  7. Add a trigger on a GitHub event, a ServiceNow incident or a new alert group.