2. Triage Each New Alert Group with an Agent
When an alert wakes someone up, the first minutes go to the same questions: what fired, which service it is, and where to look first. In this page, you build an agent that writes those answers. Then, a trigger starts it for each new alert group with severity critical or high.
Before you start
Do page 1 first. You need the test alert group and the webhook URL from that page.
Step 1: Create the Alert Triage agent
Go to Agent Studio → Agents and create an agent. In Agent name, type Alert Triage and click Create blank. Autoheal gives it the ID alert-triage.
In the editor, replace the content of agent.yaml with this:
apiVersion: agents.autoheal.ai/v1
name: Alert Triage
icon: bell-ringing
description: >-
Reads one alert group and writes what fired, the likely service and the
first three things to check. Give it the alert group ID or link.
instructions: ./instructions.md
inputs:
prompt:
description: The alert group ID, or a link to the alert group.
model:
name: claude-haiku-5-5
max_tokens: 4096
limits:
model_requests: 15
timeout: 5m
system_tools: name == "get_alert_group" || name == "get_alerts"
integration_tools: "false"
Put this in instructions.md. If the file does not exist, create it with New → New file.
You do the first triage of one alert group for the engineer on call.
1. The prompt has the alert group ID, or a link that ends with the ID. Read the alert group with `get_alert_group`.
2. Read the alerts in the group with `get_alerts`.
3. Use only the data in the alert group and its alerts. Do not guess a cause.
Write a short note with these three headings:
- **What fired**: the alert title, the severity, the source and the time of the first alert.
- **Likely service**: the service from the `service` label. If there is no `service` label, name the best candidate and say that it is a guess.
- **First three checks**: three numbered checks that the engineer can do in five minutes, with the most likely cause first.
Do not change the alert group. Do not send messages.
Commit your changes.
get_alert_group and get_alerts are system tools. They read alert groups and alerts in the project. The system_tools filter gives the agent only these two tools, and integration_tools: "false" gives it no integration tools. The agent can read alerts, but it cannot post to Slack or page anyone.
Step 2: Run the agent on the test alert group
Open the alert group from page 1 under Work → Alerts, and click Run agent. In Run an agent, find Alert Triage and select it. The prompt already names the alert group and has a link to it. Click Start run.
When the run ends, read the Summary. Check that it has the three headings, that Likely service is checkout, and that the checks are about checkout latency. In the Transcript, find the calls to get_alert_group and get_alerts.
Go back to the alert group page. Check that the Runs section lists the run.
Step 3: Create the trigger
Go to Agent Studio → Triggers, click New trigger, and select On an event. Replace the content of the trigger editor with this definition:
apiVersion: triggers.autoheal.ai/v1
name: Triage new alert groups
on:
alertgroup.created:
- anyOf:
- severity: critical
- severity: high
run:
agent: agents:alert-triage
with:
prompt: ${{ event.data.group_id }}
limits:
max_con: 3
rate_per_min: 10
| Part | What it does |
|---|---|
alertgroup.created | The trigger starts when Autoheal opens a new alert group. |
anyOf | At least one filter in the list must match. Here, the severity is critical or high. Medium and low alert groups start no run. |
prompt | The agent gets the ID of the new alert group. |
limits.max_con | At most 3 runs of this trigger at the same time. |
limits.rate_per_min | At most 10 new runs in one minute. |
The limits matter for alerts. One failure can open many alert groups in a short time. Autoheal drops the events over a limit. It does not queue them.
Step 4: Test the trigger with a sample event
In the Test fire panel, keep Schema sample and run the test. The sample is a critical alert group. Check that the Dry run result shows Match, and that the prompt input is an alert group ID. The test does not start a run.
Click Save.
Step 5: Send a second alert
Send a new test alert with severity critical. It has a new id, so Autoheal opens a new alert group:
curl -X POST "$WEBHOOK_URL" \
-H "Content-Type: application/json" \
-d '{
"id": "gs-test-checkout-errors",
"title": "gs-test: checkout 5xx rate above 5 percent",
"severity": "critical",
"status": "firing",
"service": "checkout",
"description": "5xx rate of POST /checkout is 7 percent for 5 minutes."
}'
Go to Work → Runs. A run of Alert Triage starts a few seconds later, and it shows that the trigger started it. Check that its Summary names the alert gs-test: checkout 5xx rate above 5 percent with severity critical.
Step 6: Clean up
When you finish, delete or disable the trigger, so that real alerts do not start runs before you are ready. Send each test alert again with "status": "resolved" to close its alert group. If you do not need the Custom Alert Source, delete it under Settings → Integrations. Autoheal keeps the closed alert groups.
What to know
- The trigger starts one run for each new alert group, not for each alert. An alert that joins an open alert group does not start a run.
- Other useful fields on
alertgroup.createdaretitle,status,alert_countandgroup_labels. For example, the filtergroup_labels.service: checkoutkeeps the trigger on one service. get_alert_groupandget_alertsread only what the alert source sent. For the cause, the agent needs more tools, for example metrics, logs or the Context Graph. Add them one at a time withsystem_toolsandintegration_tools. See chapter 4 of Build Your First Agent.- If Investigate Group is on under Settings → Alert response, Autoheal also starts its own investigation for each new alert group. Your trigger runs in addition to it.
Use the ready-made incident agents
The public library autohealai/agents has complete agents for on-call work:
| Agent | What it does |
|---|---|
alert-triage | Links alerts that fire together through the service dependency graph, and pages the owner of the cause one time. |
incident-response | Investigates an incident from end to end, records a root cause, and has subagents and artifact templates. |
runbook-execution | Runs the approved runbook for a known alert one time, then checks recovery. |
There is no one-click install. Copy the agent's files into your project, or clone the repository and push the agent with autoheal agents push. The library alert-triage has the same ID as your agent, so push it under a different folder name. Read its agent.yaml before you run it: these agents can use write tools, for example to page through PagerDuty.
To learn how the full versions work, see Alert Triage and Incident Response.
From the CLI
autoheal agents push agents/alert-triage -m "Add Alert Triage"
autoheal run alert-triage "<alert group ID>" --follow
autoheal triggers catalog --event alertgroup.created --sample > ev.json
autoheal triggers dry-run -f trigger.yaml --event ev.json
autoheal triggers create -f trigger.yaml
autoheal triggers runs triage-new-alert-groups
autoheal triggers delete triage-new-alert-groups --yes
triggers runs lists each firing of the trigger, and the run that it started. triggers fire also starts a real run with an event file. The sample from autoheal triggers catalog names an alert group that does not exist in your project, so fire it with an event that has a real group_id.