Skip to main content

Runbook Execution

When a known alert fires, runs its approved runbook once, then checks recovery against the service's SLO for 15 minutes before resolving. If the service has not recovered, it escalates with a log of every action.

How it works​

  1. Runs when: an alert fires that has an approved runbook.
  2. Checks it may act: reads the service's tier and runs an action only if that tier pre-approves it. Otherwise it asks, with the evidence and the check it will run afterwards.
  3. Acts once: runs the runbook's action a single time.
  4. Verifies recovery: watches error rate and p99 on the SLO's endpoint for 15 minutes. Pods being ready is not recovery.
  5. You get: a resolved incident with the action log posted to the owning team, or an escalation to on-call with what it saw and what it would check next. A restart that only postpones a leak also gets a follow-up ticket.

What it needs​

  • PagerDuty: read and resolve incidents, escalate
  • Kubernetes or AWS: run the runbook's action
  • Slack: post the action log
  • Jira (optional): file follow-ups