Skip to main content

How an Investigation Runs

This page follows one Incident Response run, an investigation, to show how the context plane, control plane and self-improvement loop come together.

The Investigation Agent​

The Investigation Agent is the orchestrator. It uses a team of subagents to:

  1. Assess the situation. Read the alert, ticket, or user query to understand the problem.
  2. Plan the investigation. Determine which tools to query and in what order.
  3. Gather evidence. Query integrations (Datadog, Grafana, GitHub, and others) through MCP.
  4. Consult the ECG. Check past incidents, skills, and service ownership for context.
  5. Form hypotheses. Generate ranked probable causes with supporting evidence.
  6. Propose fixes. Suggest remediations based on skills, past resolutions, or code-level changes.
  7. Capture learnings. After resolution, record memories that feed back into the ECG.

The agent maintains full conversation context, so investigations are interactive rather than fire-and-forget. You can ask follow-up questions, redirect it mid-run ("I think this is related to the 3 PM migration"), or expand scope without losing prior context.

Fast vs Deep investigation​

You choose how much reasoning the agent applies:

  • Fast. A single hypothesis, for a faster time-to-root-cause. Best for well-understood alerts and routine checks.
  • Deep. Multiple hypotheses, cross-validated against the evidence before a conclusion is presented. Best for ambiguous or high-stakes incidents.

Both modes produce the same evidence and run through the same adversarial review. The difference is how many hypotheses the agent explores. The Alert Triage and Incident Response agents map onto these modes by default. Alert Triage uses Fast, which suits high alert volume. Incident Response uses Deep to drive a declared incident to a validated root cause.

Evidence​

Every decision an agent makes is documented as evidence, a transparent record of its reasoning:

ElementDescription
Data sources queriedWhich tools were consulted and what data was retrieved
Hypotheses consideredAll probable causes evaluated, not just the final answer
Support for/againstWhat data supported or contradicted each hypothesis
Path decisionsWhy the agent pursued certain diagnostic paths over others
Confidence levelsHow confident the agent is in each finding, and why

Evidence makes a run auditable. It lets engineers verify reasoning rather than just results. When an agent is wrong, it shows you where the reasoning went off-track. In regulated environments it's also the documentation trail.

Adversarial review​

Before findings reach your team, a separate validation step challenges them. The step:

  • Verifies that every finding is backed by concrete, retrievable evidence.
  • Flags gaps the investigation should have covered.
  • Checks that the causal chain from symptoms to root cause holds up.
  • Filters correlations that don't imply causation.

Conclusions that survive reach you with their evidence intact. Conclusions that don't survive are sent back for more investigation. The step is there to reduce false positives, so a stated root cause comes with evidence your team can check before acting.

Alert normalization​

One incident typically fires alerts across many tools at once: a PagerDuty page, a Datadog monitor, a Grafana alert, and a Sentry spike. Rather than opening an investigation per alert, Autoheal:

  • Normalizes the alerts into a consistent format.
  • Deduplicates alerts that describe the same issue.
  • Categorizes each alert by service and failure domain.
  • Correlates simultaneous alerts to tell cascading failures from independent ones.

Your on-call engineer sees one consolidated investigation instead of a wall of noise.

Root cause and fixes​

Investigations produce a structured 5-Why RCA built from actual evidence rather than recollection. The RCA contains:

  • Incident summary
  • Timeline
  • The 5-Why chain from symptom to systemic cause
  • Impact assessment
  • Mitigations applied
  • Preventive measures with owners

Because it's generated from the run, every incident gets a post-mortem instead of only the ones someone remembers to write.

When the root cause is a code defect, the agent pinpoints the file, function, and logic responsible, then proposes the specific change. Your team reviews it with full context on why it was proposed and how it connects to the incident.

Investigation flow​

  1. Trigger

    A run starts from a PagerDuty alert, Datadog monitor, Sentry exception, Slack message, support ticket, or manual input in the Autoheal UI.

  2. Context Loading

    The agent loads relevant context from the ECG: service ownership, related skills, past incidents for this service, recent deployments, and known dependencies.

  3. Evidence Gathering

    The agent queries your connected integrations, pulling metrics, logs, traces, deployment history, and error data. It builds a timeline of events.

  4. Hypothesis Formation

    Based on evidence and ECG context, the agent generates ranked hypotheses, each recording its reasoning and supporting data.

  5. Adversarial Review

    Findings are validated to ensure they're evidence-backed and logically sound.

  6. Fix Proposals

    The agent proposes mitigations such as rollbacks, restarts, config changes, and scaling actions. For code-level issues, it surfaces preventive fixes for review.

  7. Learning

    After resolution, memories are captured and fed back into the ECG. The next run on this service then starts from more context than this one did.

That last step is the platform closing its own loop. The run that just completed becomes a memory the next one reads. The Evaluator scores it once its downstream outcome is known.