Skip to main content

2. Triage Each New Alert Group with an Agent

When an alert wakes someone up, the first minutes go to the same questions: what fired, which service it is, and where to look first. In this page, you build an agent that writes those answers. Then, a trigger starts it for each new alert group with severity critical or high.

Before you start​

Do page 1 first. You need the test alert group and the webhook URL from that page.

Step 1: Create the Alert Triage agent​

Go to Agent Studio → Agents and create an agent. In Agent name, type Alert Triage and click Create blank. Autoheal gives it the ID alert-triage.

In the editor, replace the content of agent.yaml with this:

YAML
apiVersion: agents.autoheal.ai/v1
name: Alert Triage
icon: bell-ringing
description: >-
Reads one alert group and writes what fired, the likely service and the
first three things to check. Give it the alert group ID or link.
instructions: ./instructions.md
inputs:
prompt:
description: The alert group ID, or a link to the alert group.
model:
name: claude-haiku-5-5
max_tokens: 4096
limits:
model_requests: 15
timeout: 5m
system_tools: name == "get_alert_group" || name == "get_alerts"
integration_tools: "false"

Put this in instructions.md. If the file does not exist, create it with New → New file.

MARKDOWN
You do the first triage of one alert group for the engineer on call.

1. The prompt has the alert group ID, or a link that ends with the ID. Read the alert group with `get_alert_group`.
2. Read the alerts in the group with `get_alerts`.
3. Use only the data in the alert group and its alerts. Do not guess a cause.

Write a short note with these three headings:

- **What fired**: the alert title, the severity, the source and the time of the first alert.
- **Likely service**: the service from the `service` label. If there is no `service` label, name the best candidate and say that it is a guess.
- **First three checks**: three numbered checks that the engineer can do in five minutes, with the most likely cause first.

Do not change the alert group. Do not send messages.

Commit your changes.

get_alert_group and get_alerts are system tools. They read alert groups and alerts in the project. The system_tools filter gives the agent only these two tools, and integration_tools: "false" gives it no integration tools. The agent can read alerts, but it cannot post to Slack or page anyone.

Step 2: Run the agent on the test alert group​

Open the alert group from page 1 under Work → Alerts, and click Run agent. In Run an agent, find Alert Triage and select it. The prompt already names the alert group and has a link to it. Click Start run.

When the run ends, read the Summary. Check that it has the three headings, that Likely service is checkout, and that the checks are about checkout latency. In the Transcript, find the calls to get_alert_group and get_alerts.

Go back to the alert group page. Check that the Runs section lists the run.

Step 3: Create the trigger​

Go to Agent Studio → Triggers, click New trigger, and select On an event. Replace the content of the trigger editor with this definition:

YAML
apiVersion: triggers.autoheal.ai/v1
name: Triage new alert groups
on:
alertgroup.created:
- anyOf:
- severity: critical
- severity: high
run:
agent: agents:alert-triage
with:
prompt: ${{ event.data.group_id }}
limits:
max_con: 3
rate_per_min: 10
PartWhat it does
alertgroup.createdThe trigger starts when Autoheal opens a new alert group.
anyOfAt least one filter in the list must match. Here, the severity is critical or high. Medium and low alert groups start no run.
promptThe agent gets the ID of the new alert group.
limits.max_conAt most 3 runs of this trigger at the same time.
limits.rate_per_minAt most 10 new runs in one minute.

The limits matter for alerts. One failure can open many alert groups in a short time. Autoheal drops the events over a limit. It does not queue them.

Step 4: Test the trigger with a sample event​

In the Test fire panel, keep Schema sample and run the test. The sample is a critical alert group. Check that the Dry run result shows Match, and that the prompt input is an alert group ID. The test does not start a run.

Click Save.

Step 5: Send a second alert​

Send a new test alert with severity critical. It has a new id, so Autoheal opens a new alert group:

BASH
curl -X POST "$WEBHOOK_URL" \
-H "Content-Type: application/json" \
-d '{
"id": "gs-test-checkout-errors",
"title": "gs-test: checkout 5xx rate above 5 percent",
"severity": "critical",
"status": "firing",
"service": "checkout",
"description": "5xx rate of POST /checkout is 7 percent for 5 minutes."
}'

Go to Work → Runs. A run of Alert Triage starts a few seconds later, and it shows that the trigger started it. Check that its Summary names the alert gs-test: checkout 5xx rate above 5 percent with severity critical.

Step 6: Clean up​

When you finish, delete or disable the trigger, so that real alerts do not start runs before you are ready. Send each test alert again with "status": "resolved" to close its alert group. If you do not need the Custom Alert Source, delete it under Settings → Integrations. Autoheal keeps the closed alert groups.

What to know​

  • The trigger starts one run for each new alert group, not for each alert. An alert that joins an open alert group does not start a run.
  • Other useful fields on alertgroup.created are title, status, alert_count and group_labels. For example, the filter group_labels.service: checkout keeps the trigger on one service.
  • get_alert_group and get_alerts read only what the alert source sent. For the cause, the agent needs more tools, for example metrics, logs or the Context Graph. Add them one at a time with system_tools and integration_tools. See chapter 4 of Build Your First Agent.
  • If Investigate Group is on under Settings → Alert response, Autoheal also starts its own investigation for each new alert group. Your trigger runs in addition to it.

Use the ready-made incident agents​

The public library autohealai/agents has complete agents for on-call work:

AgentWhat it does
alert-triageLinks alerts that fire together through the service dependency graph, and pages the owner of the cause one time.
incident-responseInvestigates an incident from end to end, records a root cause, and has subagents and artifact templates.
runbook-executionRuns the approved runbook for a known alert one time, then checks recovery.

There is no one-click install. Copy the agent's files into your project, or clone the repository and push the agent with autoheal agents push. The library alert-triage has the same ID as your agent, so push it under a different folder name. Read its agent.yaml before you run it: these agents can use write tools, for example to page through PagerDuty.

To learn how the full versions work, see Alert Triage and Incident Response.

From the CLI​

BASH
autoheal agents push agents/alert-triage -m "Add Alert Triage"
autoheal run alert-triage "<alert group ID>" --follow

autoheal triggers catalog --event alertgroup.created --sample > ev.json
autoheal triggers dry-run -f trigger.yaml --event ev.json
autoheal triggers create -f trigger.yaml
autoheal triggers runs triage-new-alert-groups
autoheal triggers delete triage-new-alert-groups --yes

triggers runs lists each firing of the trigger, and the run that it started. triggers fire also starts a real run with an event file. The sample from autoheal triggers catalog names an alert group that does not exist in your project, so fire it with an event that has a real group_id.

See Events, Filters and Limits in the Trigger reference.