Alert Triage
FeaturedWhen an alert group fires, links the alerts that fired with it through the service dependency graph, pages the owner of the cause once, and proposes tuning for noisy monitors.
How it works
- Runs when: a new alert group fires.
- Collects the window: every alert that fired within ten minutes, from Datadog and Alertmanager, and each service's owner from the catalog.
- Finds the cause: follows slow or failing edges in the dependency graph to the service that is unhealthy on its own terms. Callers that are only waiting, or only taking retries, are linked, not blamed.
- Pages once: the owner of the cause, with every linked alert attached. Owners of linked services are notified, not paged.
- Tunes noise: a monitor that fires often, resolves itself and is never acknowledged gets a threshold proposal, backtested against 30 days. It is never muted, and the change waits for the monitor owner.
What it needs
- Datadog or Prometheus: read alerts and the service graph; update monitors with approval
- PagerDuty: page the owning team