Prometheus Integration

Connect Prometheus to let agents run PromQL queries and discover metric and label metadata directly during investigations — no Grafana required. You can also forward Prometheus Alertmanager alerts to Autoheal through a webhook. Autoheal ingests and groups these alerts, and makes them available for triage on the Alerts page.

This integration works with Prometheus and Prometheus-API-compatible backends (Thanos, Cortex, Grafana Mimir, VictoriaMetrics).

Capabilities

Once connected, agents can:

Capability	Description
Query Metrics	Run PromQL instant and range queries (prometheus_query). The step is auto-selected from the time window when you don't set one
Discover Metrics & Labels	List label names and values, find series by selector, and read metric type/help/unit metadata (prometheus_metadata) so agents build correct queries without guessing
Test Connection	Verify the endpoint and credentials, and report the Prometheus version (prometheus_test_connection)
Alert Source Webhooks	Receive Alertmanager firing and resolved alerts via webhook, with all labels and annotations preserved for grouping and routing
Multiple Environments	Add one instance per environment (prod, stage, dev, DR). Agents pick the right one from its name during an investigation

Prerequisites

• A Prometheus HTTP API endpoint reachable by Autoheal (see Private and on-prem access if it is not internet-reachable)
• Credentials matching how your Prometheus is protected: none (anonymous), basic auth, or a bearer token. (Prometheus is unauthenticated by default and is usually protected by a reverse proxy.)
• For multi-tenant backends (Mimir/Cortex): your tenant / Scope OrgID

Setup

1

Add the Integration in Autoheal

1. Go to Integrations in Autoheal and click Prometheus
2. Enter a name that identifies the environment — e.g. Prometheus – prod. The name is what agents use to pick the right instance, so make it unambiguous.

2

Enter the Endpoint

• Prometheus Endpoint: the base URL of the HTTP API, e.g. http://prometheus.monitoring.svc:9090. Do not include /api/v1.
• For Mimir/Cortex, use the query base URL including any path prefix, e.g. https://mimir.internal/prometheus, and set Scope OrgID to your tenant.

3

Choose Authentication

Pick the auth type that matches your endpoint:

• None (anonymous) — no credentials (e.g. network-segmented or behind a connector)
• Username & Password — HTTP basic auth
• Bearer Token — a token sent as Authorization: Bearer <token> (e.g. a reverse proxy or Kubernetes ServiceAccount token)

4

Test and Save

Click Test Connection to check the endpoint and credentials, then Save.

note

Multiple instances. Add a separate Prometheus integration for each environment (for example Prometheus – prod, Prometheus – stage, Prometheus – dev, Prometheus – DR). When several exist, an agent chooses the one whose name matches the environment of the alert it is investigating. If it cannot tell, it asks you to disambiguate.

Authentication Options

Auth type	When to use	Fields
None	Prometheus is reachable without credentials	—
Username & Password	Basic auth (e.g. nginx / oauth2-proxy in front)	Username, Password
Bearer Token	Reverse-proxy or ServiceAccount bearer token	Bearer Token

tip

Give Autoheal read-only access. Prometheus's query and metadata endpoints are read-only, and this integration never calls admin endpoints (no delete_series, snapshots, or writes).

Example Queries

Once connected, you can ask an agent questions like:

What's the p99 latency for the checkout service over the last hour?

Show CPU usage for the nodes in the prod cluster for the last 6 hours

Is any pod in the payments namespace OOMKilling? Check container_memory_working_set_bytes vs limits

Which jobs are currently down? (up == 0)

List the label values for "job" so I know what's being scraped

Alert Source Setup

Prometheus Alertmanager can forward alerts to Autoheal via a webhook receiver. Autoheal preserves all labels and annotations, groups alerts, and auto-resolves them when they clear.

1

Enable the Webhook in Autoheal

1. Open your Prometheus integration in Autoheal
2. Scroll to the Alert Source section and copy the Webhook URL (it already contains a unique per-integration secret)
3. Optionally, enter a Webhook Signing Secret for a Bearer-token check on every delivery

2

Add a Webhook Receiver in Alertmanager

In your Alertmanager configuration, add a webhook receiver whose url is the copied Webhook URL:

receivers:
  - name: autoheal
    webhook_configs:
      - url: "<paste the Autoheal Webhook URL here>"
        send_resolved: true
        # Only if you set a Webhook Signing Secret in Autoheal:
        http_config:
          authorization:
            type: Bearer
            credentials: "<the same signing secret>"

warning

Keep send_resolved: true so Autoheal auto-resolves alerts when they clear. If you set a Webhook Signing Secret in Autoheal, the receiver must send the matching Authorization: Bearer header. Otherwise, Autoheal rejects deliveries with 401. If you leave the secret blank, that check is disabled. The unguessable URL is then the only guard.

3

Route Alerts to the Receiver

In your Alertmanager route tree, send the alerts you want triaged to the autoheal receiver:

route:
  routes:
    - receiver: autoheal
      matchers:
        - severity=~"warning|critical"
      continue: true

4

Test the Integration

Reload Alertmanager and fire a test alert. It should appear on Autoheal's Alerts page within a few seconds.

note

A still-firing alert carries endsAt: "0001-01-01T00:00:00Z". Autoheal treats this as "no end time yet" and keeps the alert open. Every alert must carry an alertname label (Prometheus alerting rules always set one).

Private and on-prem access

note

This applies to Autoheal's SaaS deployment. In a BYOC deployment, the stack runs inside your own network and reaches Prometheus directly by endpoint. So the Private Access connection field is not shown.

Query routing over a Private Access connection runs inside the agent sandbox and is enabled per organization. Contact support to enable it. See Private Access.

Alert delivery still works when your Prometheus is on a private network and not reachable from the internet. Alertmanager makes an outbound call to the Webhook URL. But queries need a path into your network:

1

Add a Private Access connection

Follow the Tailscale guide to add a connection (Tailscale) and join the host or subnet running Prometheus to your private network.

2

Bind Prometheus to it

In the Prometheus integration, set Private Access connection to the connection you created. Queries for this instance then route over your private network to the endpoint.

note

Only integrations you explicitly bind to a Private Access connection route through it. Everything else continues to reach public endpoints directly. Point the endpoint at a URL whose TLS certificate your CA trusts, or use http:// inside the tunnel (traffic over your private network is already encrypted). Self-signed HTTPS on the private path is not supported yet.

Troubleshooting

401 / 403 on Test Connection

• Check that the Auth type matches how your Prometheus is protected (None vs Basic vs Bearer)
• For Bearer, ensure the token is current and sent as Authorization: Bearer <token> by your proxy
• For Basic, re-enter the username and password (watch for trailing spaces)

Connection timeout / cannot reach endpoint

• Check that the Endpoint is the Prometheus HTTP API base URL and does not include /api/v1
• If Prometheus is on a private network, add and bind a Private Access connection (SaaS deployments). See Private and on-prem access

No data returned

• Check the PromQL syntax and that the time range includes data
• Use prometheus_metadata (ask an agent to list labels/series) to verify the metric and label names exist
• For Mimir/Cortex, verify Scope OrgID matches the tenant that holds the data

Query is slow or times out

• Narrow the time range, or let the agent use a coarser step for wide ranges
• Reduce cardinality (aggregate with sum by (...) rather than returning every series)

Webhook alerts not appearing

• Check that the receiver url in Alertmanager exactly matches the Webhook URL in Autoheal
• Check that the alerts are routed to the autoheal receiver in your route tree
• Check that the webhook is enabled in the Autoheal integration settings

Webhook returns 401 Unauthorized

• If you set a Webhook Signing Secret in Autoheal, set the same value as authorization.credentials (type Bearer) in the receiver's http_config
• If you don't want Bearer auth, clear the Webhook Signing Secret field in the Alert Source section

Resolved alerts never arrive

• Check that send_resolved: true is set on the webhook receiver