# AI Incident Runbook
What to do when the model is on fire.

_A production runbook for AI incidents. Covers detection, severity, on-call steps, internal and customer comms, and the postmortem trigger._

Used in: Lesson expert/05 — https://vetslearn.com/lessons/expert/05
Source: https://vetslearn.com/resources/templates/engineering#t-incident-runbook

---

# Incident Runbook: [SYSTEM NAME]

**On-call rotation:** [ON-CALL ROTATION]
**Incident channel:** [INCIDENT CHANNEL]
**Status page:** [STATUS PAGE URL]

---

## 1. Detection signals

*Fill this in: how you know something is wrong. List the dashboards, alerts, and channels.*

- Latency alert: p95 over [N] seconds for [M] minutes
- Error rate alert: 5xx over [N]% for [M] minutes
- Refusal rate alert: refusals over [N]% baseline
- Quality alert: eval pass rate drops over [N]%
- Customer reports in [INCIDENT CHANNEL]
- Social media or support ticket spike

## 2. Severity matrix

| Sev | Definition | Response time | Who pages |
|---|---|---|---|
| SEV-1 | System down, data leak, or wrong answer at scale to paying users | 15 min | On-call + eng lead + product lead |
| SEV-2 | Degraded quality or latency affecting most users | 30 min | On-call + eng lead |
| SEV-3 | Affects a subset of users or a non-critical path | 2 hours | On-call |
| SEV-4 | Cosmetic or internal-only | Next business day | Ticket queue |

## 3. On-call playbook

**Step 1 — Acknowledge.** Page in [INCIDENT CHANNEL] within 5 min. State sev, scope, and "investigating."

**Step 2 — Triage.** Check dashboards: latency, error rate, refusal rate, cost. Check vendor status pages. Check recent deploys in the last 24 hours.

**Step 3 — Contain.** If a recent deploy looks bad, roll back. If a vendor is down, flip the fallback model. If prompt injection, block the offending pattern at the gateway.

**Step 4 — Communicate.** Post internal update every 30 min. Update status page for SEV-1 and SEV-2.

**Step 5 — Resolve.** Confirm metrics returned to baseline for 30 min. Post resolution note.

**Step 6 — Schedule postmortem.** Within 24 hours for SEV-1 and SEV-2.

## 4. Communication templates

### Internal update

```
[SEV-X] [SYSTEM NAME] — [short title]
Status: investigating | identified | mitigating | resolved
Scope: [who is affected, how many, since when]
Current action: [what we are doing right now]
Next update: [time]
IC: [name]
```

### Customer-facing status page

```
We are investigating reports of [problem] with [SYSTEM NAME]. Some users may see [symptom]. We will post an update within [time].
```

Resolution:

```
The issue with [SYSTEM NAME] is resolved as of [time]. Cause: [one sentence]. We will publish a full writeup within 5 business days.
```

## 5. Postmortem trigger

Auto-trigger a blameless postmortem for:
- Any SEV-1 or SEV-2
- Any incident that hit customer trust (wrong answer in public, data exposure)
- Any incident that required a rollback
- Any near-miss caught by a human, not the alert system

Postmortem owner: the incident commander. Due: 5 business days. Template: [link to postmortem template].

---

## How to use

Print this. Pin it in the on-call channel. Walk through it in a tabletop drill before you ship — first-time incidents are not the time to read the runbook. Update it after every postmortem with what was missing.

## Check before you use it

- Every alert threshold matches an alert that exists in your monitoring today.
- Page the on-call rotation once as a drill. A runbook with a dead pager is paper.
- Test the rollback and the fallback model path before the first incident, not during it.
- Run one tabletop walkthrough before you ship.