Grafana Integration
Connect Grafana to enable the AI agent to query dashboards, metrics, and alerts during investigations. You can also forward Grafana alerts to Autoheal via webhook so alerts are ingested, grouped, and available for triage in the Alerts page.
Capabilities
Once connected, the AI agent can:
| Capability | Description |
|---|---|
| Query Logs | Query logs from Loki or VictoriaLogs data sources |
| Query Metrics | Fetch data from Prometheus and other metric sources |
| View Dashboards | Access dashboard panels and visualizations |
| Check Alerts | Review alert rules and their current state |
| Alert Source Webhooks | Receive Grafana firing and resolved alerts via webhook, including batched alert payloads |
| Search Annotations | Find annotations marking events |
Supported Log Backends
The Grafana integration auto-detects your log backend and uses the appropriate query tools:
| Backend | Datasource Type | Query Language | Auto-Detected |
|---|---|---|---|
| Grafana Loki | loki | LogQL | Yes |
| VictoriaLogs | victoriametrics-logs-datasource | LogsQL | Yes |
No additional configuration is needed — the agent discovers available data sources automatically.
Prerequisites
- A Grafana instance (Cloud or self-hosted)
- Grafana 9+ with Unified Alerting enabled (required for alert-source webhooks)
- A Service Account token or a legacy API key
- Viewer or higher permissions
Setup
Both service account tokens (recommended) and legacy API keys are supported.
Option A: Service Account Token (Recommended)
- Log in to your Grafana instance
- Navigate to Administration → Service accounts
- Click Add service account
- Name it (e.g., "Autoheal Integration")
- Assign the Viewer role
- In the service account, click Add token
- Give it a name and set expiration (or no expiration)
- Copy the generated token immediately
- Go to Integrations in Autoheal
- Click Grafana
- Enter a name (e.g., "Production Grafana")
Enter the following:
- URL: Your Grafana URL (e.g.,
https://your-org.grafana.net) - API Token: The service account token you created
Click Test Connection to verify, then Save.
Option B: Legacy API Key
Legacy API keys are deprecated in Grafana 9+ but still supported by this integration.
- Log in to your Grafana instance
- Navigate to Configuration → API keys
- Click Add API key
- Name it (e.g., "Autoheal Integration")
- Set the role to Viewer (or higher)
- Set expiration as needed
- Copy the generated key immediately — it will not be shown again
- Go to Integrations in Autoheal
- Click Grafana
- Enter a name (e.g., "Production Grafana")
Enter the following:
- URL: Your Grafana URL (e.g.,
https://grafana.your-company.com) - API Token: The legacy API key you created
Click Test Connection to verify, then Save.
Grafana is deprecating legacy API keys in favor of service account tokens. If your Grafana version supports service accounts, we recommend using Option A.
Required Permissions
The service account or API key needs:
| Permission | Why It's Needed |
|---|---|
| Viewer role | Basic read access to dashboards and data |
| Data source access | Query underlying data sources |
For self-hosted Grafana, ensure the service account or API key has access to the specific data sources you want to query.
Grafana Cloud vs Self-Hosted
Grafana Cloud
Use your Grafana Cloud URL:
https://your-org.grafana.net
Self-Hosted Grafana
Use your Grafana server URL:
https://grafana.your-company.com
Ensure the URL is accessible from Autoheal (check firewall rules).
Example Queries
Once connected, you can ask the AI agent:
Show me the request rate from the API dashboard
What alerts are currently firing in Grafana?
Get CPU metrics for the web servers from the last 30 minutes
Grafana Version Compatibility
The integration automatically handles different Grafana API versions. Grafana 11.x introduced changes to datasource proxy endpoints that are transparently managed by the agent.
Version Detection
The agent can detect your Grafana version to optimize API calls:
What version of Grafana are we using?
This returns version information and API compatibility mode.
Automatic Fallback
The integration uses version-aware endpoint selection with automatic fallback:
- Grafana 11.x and later: Uses ID-based datasource proxy (primary), falls back to UID-based if needed
- Grafana 10.x and earlier: Uses UID-based datasource proxy (primary), falls back to ID-based if needed
This ensures compatibility across different Grafana versions without requiring manual configuration.
Alert Source Setup
Grafana can also act as an alert source by forwarding alerts to Autoheal via webhook using Grafana Unified Alerting (Grafana 9+).
Configure Grafana as an Alert Source
- Open your Grafana integration in Autoheal
- Go to the Alert Source tab
- Enable the webhook and copy the Webhook URL
- Optionally, enter a Webhook Signing Secret for additional authentication
- In Grafana, go to Alerting → Contact points
- Click Create contact point
- Name it anything for your own organization (for example, "Autoheal")
- Choose Webhook as the contact point type
- Paste the Webhook URL into the URL field
- Ensure HTTP Method is POST (not PUT)
- If you set a Webhook Signing Secret in Autoheal, expand Optional Webhook settings, set Authorization Header - Scheme to
Bearer, and paste the same secret into Authorization Header - Credentials - Save the contact point
- Go to Alerting → Notification policies
- Edit the default policy or create a new one
- Set the contact point to "Autoheal"
- Save the policy
Create or trigger a test alert in Grafana to verify alerts appear in Autoheal's Alerts page. You should typically see new alert entries within seconds (timing can vary based on grouping settings).
Grafana Unified Alerting is available in Grafana 9 and later. Both Grafana Cloud and self-hosted instances are supported.
The Webhook Signing Secret is an optional second layer of authentication. The webhook URL already contains a unique secret for identifying your integration. Adding a signing secret provides additional verification via the Authorization header, ensuring only Grafana can send alerts to your endpoint.
Grafana sends both firing and resolved state changes through the same webhook endpoint, and one delivery can include multiple alerts in the alerts[] array. Resolved alerts are processed automatically and update alert status in Autoheal.
Troubleshooting
401 Unauthorized
- Verify the API token is correct and hasn't expired
- Check that the service account hasn't been disabled
- Ensure the token was copied completely
Connection Refused (Self-Hosted)
- Verify the Grafana URL is correct
- Check firewall rules allow access from Autoheal
- Ensure Grafana is running and accessible
No Dashboards Found
- Verify the service account has Viewer permissions
- Check folder permissions for dashboards
Webhook Alerts Not Appearing in Autoheal
- Verify the webhook URL in Grafana matches the URL shown in Autoheal
- Check that the contact point is assigned to a notification policy
- Ensure the alert rule is firing (check Alerting → Alert rules in Grafana)
- Verify the webhook is enabled in the Autoheal integration settings
Webhook Returns 401 Unauthorized
- If you configured a Webhook Signing Secret in Autoheal, ensure the same token is set in Grafana's Contact Point under Authorization Header - Credentials with Scheme set to Bearer
- If you don't want auth header verification, remove the Webhook Signing Secret from the Alert Source tab in Autoheal
404 Errors on Prometheus Queries
If you receive 404 errors specifically on Prometheus queries:
- The agent automatically handles version-specific API formats
- The issue may be with datasource configuration or permissions
- Verify the datasource UID is correct using the "List data sources" query
- Check that the service account has access to the Prometheus datasource
Note: Loki and VictoriaLogs queries use a different endpoint format and typically don't have this issue.