Runbook Execution
When a known alert fires, runs its approved runbook once, then checks recovery against the service's SLO for 15 minutes before resolving. If the service has not recovered, it escalates with a log of every action.
How it works
- Runs when: an alert fires that has an approved runbook.
- Checks it may act: reads the service's tier and runs an action only if that tier pre-approves it. Otherwise it asks, with the evidence and the check it will run afterwards.
- Acts once: runs the runbook's action a single time.
- Verifies recovery: watches error rate and p99 on the SLO's endpoint for 15 minutes. Pods being ready is not recovery.
- You get: a resolved incident with the action log posted to the owning team, or an escalation to on-call with what it saw and what it would check next. A restart that only postpones a leak also gets a follow-up ticket.
What it needs
- PagerDuty: read and resolve incidents, escalate
- Kubernetes or AWS: run the runbook's action
- Slack: post the action log
- Jira (optional): file follow-ups