Prometheus Integration

Connect Prometheus to let the AI agent run PromQL queries and discover metric and label metadata directly during investigations — no Grafana required. You can also forward Prometheus Alertmanager alerts to Autoheal via webhook so they are ingested, grouped, and available for triage on the Alerts page.

This integration works with Prometheus and Prometheus-API-compatible backends (Thanos, Cortex, Grafana Mimir, VictoriaMetrics).

Capabilities

Once connected, the AI agent can:

Capability	Description
Query Metrics	Run PromQL instant and range queries (prometheus_query); the step is auto-selected from the time window when you don't set one
Discover Metrics & Labels	List label names and values, find series by selector, and read metric type/help/unit metadata (prometheus_metadata) so the agent builds correct queries without guessing
Test Connection	Verify the endpoint and credentials, and report the Prometheus version (prometheus_test_connection)
Alert Source Webhooks	Receive Alertmanager firing and resolved alerts via webhook, with all labels and annotations preserved for grouping and routing
Multiple Environments	Add one instance per environment (prod, stage, dev, DR); the agent picks the right one from its name during an investigation

Prerequisites

• A Prometheus HTTP API endpoint reachable by Autoheal (see Private and on-prem access if it is not internet-reachable)
• Credentials matching how your Prometheus is protected: none (anonymous), basic auth, or a bearer token (Prometheus is unauthenticated by default and usually protected by a reverse proxy)
• For multi-tenant backends (Mimir/Cortex): your tenant / Scope OrgID

Setup

1

Add the Integration in Autoheal

1. Go to Integrations in Autoheal and click Prometheus
2. Enter a name that identifies the environment — e.g. Prometheus – prod. The name is what the agent uses to pick the right instance, so make it unambiguous.

2

Enter the Endpoint

• Prometheus Endpoint: the base URL of the HTTP API, e.g. http://prometheus.monitoring.svc:9090. Do not include /api/v1.
• For Mimir/Cortex, use the query base URL including any path prefix, e.g. https://mimir.internal/prometheus, and set Scope OrgID to your tenant.

3

Choose Authentication

Pick the auth type that matches your endpoint:

• None (anonymous) — no credentials (e.g. network-segmented or behind a connector)
• Username & Password — HTTP basic auth
• Bearer Token — a token sent as Authorization: Bearer <token> (e.g. a reverse proxy or Kubernetes ServiceAccount token)

4

Test and Save

Click Test Connection to verify the endpoint and credentials, then Save.

note

Multiple instances. Add a separate Prometheus integration for each environment (for example Prometheus – prod, Prometheus – stage, Prometheus – dev, Prometheus – DR). When several exist, the agent chooses the one whose name matches the environment of the alert it is investigating, and will ask you to disambiguate if it cannot tell.

Authentication Options

Auth type	When to use	Fields
None	Prometheus is reachable without credentials	—
Username & Password	Basic auth (e.g. nginx / oauth2-proxy in front)	Username, Password
Bearer Token	Reverse-proxy or ServiceAccount bearer token	Bearer Token

tip

Give Autoheal read-only access. Prometheus's query and metadata endpoints are read-only, and this integration never calls admin endpoints (no delete_series, snapshots, or writes).

Example Queries

Once connected, you can ask the AI agent questions like:

What's the p99 latency for the checkout service over the last hour?

Show CPU usage for the nodes in the prod cluster for the last 6 hours

Is any pod in the payments namespace OOMKilling? Check container_memory_working_set_bytes vs limits

Which jobs are currently down? (up == 0)

List the label values for "job" so I know what's being scraped

Alert Source Setup

Prometheus Alertmanager can forward alerts to Autoheal via a webhook receiver. Autoheal preserves all labels and annotations, groups alerts, and auto-resolves them when they clear.

1

Enable the Webhook in Autoheal

1. Open your Prometheus integration in Autoheal
2. Scroll to the Alert Source section and copy the Webhook URL (it already contains a unique per-integration secret)
3. Optionally, enter a Webhook Signing Secret for a Bearer-token check on every delivery

2

Add a Webhook Receiver in Alertmanager

In your Alertmanager configuration, add a webhook receiver whose url is the copied Webhook URL:

receivers:
  - name: autoheal
    webhook_configs:
      - url: "<paste the Autoheal Webhook URL here>"
        send_resolved: true
        # Only if you set a Webhook Signing Secret in Autoheal:
        http_config:
          authorization:
            type: Bearer
            credentials: "<the same signing secret>"

warning

Keep send_resolved: true so Autoheal auto-resolves alerts when they clear. If you set a Webhook Signing Secret in Autoheal, the receiver must send the matching Authorization: Bearer header or deliveries are rejected with 401. Leaving the secret blank disables that check — the unguessable URL is then the only guard.

3

Route Alerts to the Receiver

In your Alertmanager route tree, send the alerts you want triaged to the autoheal receiver:

route:
  routes:
    - receiver: autoheal
      matchers:
        - severity=~"warning|critical"
      continue: true

4

Test the Integration

Reload Alertmanager and fire a test alert. It should appear on Autoheal's Alerts page within a few seconds.

note

A still-firing alert carries endsAt: "0001-01-01T00:00:00Z" — Autoheal treats this as "no end time yet" and keeps the alert open. Every alert must carry an alertname label (Prometheus alerting rules always set one).

Private and on-prem access

If your Prometheus is on a private network and not reachable from the internet, alert delivery still works (Alertmanager makes an outbound call to the Webhook URL), but queries need a path into your network:

1

Add a Tailscale connector

Follow the Private Access guide to add a Tailscale connector and join the host or subnet running Prometheus to your tailnet.

2

Bind Prometheus to the connector

In the Prometheus integration, set Network Connector to the Tailscale connector you created. Queries for this instance then route over your tailnet to the private endpoint.

note

Only integrations you explicitly bind to a connector route through it — everything else continues to reach public endpoints directly. Point the endpoint at a URL whose TLS certificate your CA trusts (or use http:// inside the tunnel — traffic over the tailnet is already encrypted); self-signed HTTPS on the private path is not supported yet.

Troubleshooting

401 / 403 on Test Connection

• Confirm the Auth type matches how your Prometheus is protected (None vs Basic vs Bearer)
• For Bearer, ensure the token is current and sent as Authorization: Bearer <token> by your proxy
• For Basic, re-enter the username and password (watch for trailing spaces)

Connection timeout / cannot reach endpoint

• Verify the Endpoint is the Prometheus HTTP API base URL and does not include /api/v1
• If Prometheus is on a private network, add and bind a Tailscale connector (see Private and on-prem access)

No data returned

• Check the PromQL syntax and that the time range includes data
• Use prometheus_metadata (ask the agent to list labels/series) to confirm the metric and label names exist
• For Mimir/Cortex, confirm Scope OrgID matches the tenant that holds the data

Query is slow or times out

• Narrow the time range, or let the agent use a coarser step for wide ranges
• Reduce cardinality (aggregate with sum by (...) rather than returning every series)

Webhook alerts not appearing

• Confirm the receiver url in Alertmanager exactly matches the Webhook URL in Autoheal
• Confirm the alerts are routed to the autoheal receiver in your route tree
• Ensure the webhook is enabled in the Autoheal integration settings

Webhook returns 401 Unauthorized

• If you set a Webhook Signing Secret in Autoheal, set the same value as authorization.credentials (type Bearer) in the receiver's http_config
• If you don't want Bearer auth, clear the Webhook Signing Secret field in the Alert Source section

Resolved alerts never arrive

• Ensure send_resolved: true is set on the webhook receiver