The On-Call Problem

An alert fires at 2am. The on-call engineer wakes up, opens their laptop, reads the alert, tries to remember what this alert means, searches Slack for the last time it fired, finds the runbook (if one exists), follows the steps, and resolves the incident, or escalates if they cannot.

This process takes 20-90 minutes for incidents that have been seen before and resolved the same way before. The knowledge exists in the runbook. The runbook exists in Confluence. The Confluence page was last updated 14 months ago by an engineer who left the company.

AI-powered runbooks change this. Not by replacing the on-call engineer, human judgment is still required for novel incidents, but by automating the execution of known resolution paths for known incident patterns.

What Self-Healing Infrastructure Means in Practice

True self-healing infrastructure, systems that detect problems and resolve them without human intervention, is a goal that large technology companies have been building toward for years. For most engineering organizations, a more achievable near-term goal is AI-assisted incident response: systems that detect problems, diagnose them, execute the first 2-3 remediation steps automatically, and only page a human if those steps do not resolve the issue.

Tier 1: AI-assisted triage (achievable now)

When an alert fires, AI assembles the context the engineer needs before they open their laptop:

This is the Incident Triage Brief automation. It does not resolve the incident, it eliminates the first 15-20 minutes of context gathering.

Tier 2: Automated first-response actions (achievable for well-defined scenarios)

For incident types with known, safe, reversible first-response actions, AI executes those actions automatically:

Each automated action is logged, with the alert that triggered it and the outcome. A human reviews the log and approves or modifies the automation for next time.

Tier 3: Adaptive remediation (emerging, requires significant engineering investment)

The most advanced implementations use ML models trained on historical incident data to select and execute remediation actions based on incident patterns, and adjust their decision models based on outcomes. This is where the large cloud providers operate.

Building Tier 1 and Tier 2

Tier 1, The Incident Triage Brief:

1. Connect PagerDuty or Datadog to Zapier via webhook

2. When alert fires, Zapier queries GitHub for recent deployments in affected repository

3. Zapier searches incident history sheet for similar past incidents

4. ChatGPT generates structured triage brief with context and recommended first steps

5. Brief posted to Slack incident channel before on-call engineer sees the page

Tier 2, Automated first-response:

This requires more careful implementation. For each automation candidate:

1. Define the trigger condition precisely (not "high CPU" but "CPU > 85% for 5 consecutive minutes on web tier")

2. Define the action precisely (restart web tier pods via Kubernetes API)

3. Define the success condition (CPU returns below 70% within 3 minutes)

4. Define the escalation condition (if action does not resolve within 5 minutes, page on-call)

5. Log every execution with outcome

Start with one automation for your most common, most benign incident type. Measure the outcome. Expand.

What This Is Worth

Organizations that have implemented AI-assisted incident response report:

The last point is underappreciated. Building AI-assisted runbooks forces the team to write down what they currently know only in their heads. That documentation has value beyond the automation.