Skip to main content

1. Send an Alert to Autoheal

An alert source is an integration that sends alerts to Autoheal, for example Prometheus, Datadog or PagerDuty. The Custom Alert Source accepts any JSON that your system can send with an HTTP POST. In this page, you use it to send one test alert from your terminal. You do not need a monitoring tool.

An alert is one signal from a source, for example "checkout latency is high". An alert group holds one or more alerts that Autoheal treats as one problem, and agents and triggers work on alert groups.

Step 1: Make sure that a test alert reaches nobody​

A new alert group can post to Slack, page a person, or start an investigation. Do this check before you send an alert.

Go to Settings → Alert response. Check these tabs:

  • Notify: if Notifications shows Enabled with a Slack channel, each new alert group posts a message to that channel.
  • Investigate Group: if Auto-investigate new alert groups is on, Autoheal starts its own investigation of each new alert group.
  • Escalate Group: when On-Call is turned on, an escalation path can page a person for an alert that matches it.

If any of these can reach people, use a different project, or ask a Project Admin what is safe. Do not turn off settings that your team uses.

Step 2: Add a Custom Alert Source​

Go to Settings → Integrations. Under Available Integrations, click Custom Alert Source. The form opens on the Configure tab.

  1. In Name, type Test alerts.
  2. In Payload Mapping, under Alert fields, type id in Identity, title in Title and severity in Severity.
  3. Under How Autoheal knows an event is resolved, type status in Resolved status field and resolved in Resolved values.
  4. Under Labels, click Add label. Type service in Label name and service in JSON path.
  5. Keep Enable grouping off. Each test alert then gets its own alert group.

Each mapping field is a path in your JSON. Identity must be unique and stable for each alert. When the same identity comes again, Autoheal updates the same alert and does not make a new one.

Step 3: Preview the mapping and save​

Paste this sample into the Preview box and click Preview:

JSON
{
"id": "gs-test-checkout-latency",
"title": "gs-test: checkout p95 latency above 2 seconds",
"severity": "high",
"status": "firing",
"service": "checkout",
"description": "p95 latency of POST /checkout is 2.4 s for 10 minutes. The threshold is 2 s."
}

The Mapping preview dialog shows the alert that Autoheal makes. Check that the identity is gs-test-checkout-latency, the severity is high and the label service is checkout. The description field is not in the mapping, so it becomes an annotation.

In the Webhook configuration card, copy the Webhook URL. Then click Save.

The webhook URL is a password

The secret part of the URL lets anyone send alerts to your project. Do not put it in a repository or a chat. If the URL leaks, click Regenerate webhook URL.

Step 4: Send the test alert​

In a terminal, put the URL in a variable and send the alert:

BASH
export WEBHOOK_URL='<the Webhook URL>'

curl -X POST "$WEBHOOK_URL" \
-H "Content-Type: application/json" \
-d '{
"id": "gs-test-checkout-latency",
"title": "gs-test: checkout p95 latency above 2 seconds",
"severity": "high",
"status": "firing",
"service": "checkout",
"description": "p95 latency of POST /checkout is 2.4 s for 10 minutes. The threshold is 2 s."
}'

Check that the reply is {"status":"accepted"}. This reply means only that Autoheal received the JSON. Autoheal maps it after the reply.

Step 5: Find the alert group​

Go to Work → Alerts. Check that the list has a row titled gs-test: checkout p95 latency above 2 seconds, with the status firing and the severity high. Each row is an alert group.

Click the row. The alert group page shows the Summary, the Details and a Runs section. The page address ends with the alert group ID. Keep this page open for page 2.

What to know​

  • With Enable grouping off, each new identity makes its own alert group. With grouping on, Autoheal puts related alerts in one group by your rules under Settings → Alert response → Group Alerts. See Grouping.
  • To close the alert, send the same JSON with "status": "resolved". The alert group then shows as resolved.
  • If you set a Webhook signing secret, every request must also send the header Authorization: Bearer <secret>.
  • Fields that are not in the mapping become annotations only if they are top-level text, number or true/false values. Map nested fields with dotted paths, for example alert.labels.service.
  • For a real source, use its own integration if Autoheal has one, for example Prometheus or Datadog. Use the Custom Alert Source for tools that have no integration.
  • Most organizations do not show the Investigations, Incidents and On-Call pages. This track does not use them. To learn what they do, see Incident Response.

From the CLI​

BASH
autoheal alerts groups list --severity high
autoheal alerts groups get <alert group ID>
autoheal alerts groups alerts <alert group ID>

The first command lists the alert groups with severity high. The second shows one alert group, and the third lists the alerts in it. autoheal integrations create --input <file> can also add the alert source from a JSON file, but it has no preview. Use the web app for steps 2 and 3.

See Custom Alert Source for the full mapping, batches and grouping.