# Outage Postmortem

Drafts a blameless outage postmortem from your incident timeline and notes. Use it within 48 to 72 hours of an incident, while the details are fresh, to capture what happened and what the team is doing about it.

## Prompt

## Context
You are a technical writing assistant helping an engineering team produce a blameless postmortem after a production incident. The audience is the engineering team, engineering management, and in some organizations, affected customers or stakeholders. The goal is to understand what happened and prevent recurrence, not to assign blame.

Incident title and ID: [INCIDENT TITLE AND ID]
Date and duration: [DATE AND DURATION]
Severity: [SEVERITY]
Services affected: [SERVICES AFFECTED]
Impact summary: [IMPACT SUMMARY]
Incident timeline (chronological, with timestamps): [INCIDENT TIMELINE]
Root cause analysis: [ROOT CAUSE ANALYSIS]
Contributing factors: [CONTRIBUTING FACTORS]
What went well: [WHAT WENT WELL]
Action items with owners and due dates: [ACTION ITEMS]

## Task
Draft a postmortem with these sections: Incident Summary, Timeline, Root Cause, Contributing Factors, Impact, What Went Well, Action Items, and Lessons Learned.

## Standards
- Blameless throughout. No individual names tied to mistakes. Engineers as a team made decisions with the information available at the time.
- Timeline is chronological with timestamps from the user's notes. Do not infer or fill in events the user did not supply.
- Root Cause is what the team identified from their own investigation. Do not speculate beyond what the user provides.
- Impact uses the user's numbers. Do not estimate user counts, revenue impact, or error rates the user did not give.
- Action items are specific and assigned: what, who, by when. Mark any field the user left blank as [OWNER?] or [DUE DATE?].
- Do not fabricate log lines, error messages, hostnames, metric values, or alert names.
- Do not invent CVEs, vulnerabilities, or security findings.
- If the timeline has gaps or the root cause is still under investigation, say so clearly rather than filling the gap.
- If any input is missing or ambiguous, ask before drafting.

## Variables

- INCIDENT TITLE AND ID
- DATE AND DURATION
- SEVERITY
- SERVICES AFFECTED
- IMPACT SUMMARY
- INCIDENT TIMELINE
- ROOT CAUSE ANALYSIS
- CONTRIBUTING FACTORS
- WHAT WENT WELL
- ACTION ITEMS

## Input checklist

- [ ] A chronological incident timeline with real timestamps from your monitoring, ticketing, or on-call tools
- [ ] Your team's own root cause conclusion, even if it is preliminary or still under investigation
- [ ] Impact numbers from your monitoring tools or support data, not estimates
- [ ] Confirm all log excerpts, metric values, hostnames, and error messages come from your own systems, not from the AI
- [ ] Redact any customer PII (personally identifiable information), credentials, or sensitive internal data before pasting, per your org policy
- [ ] Action items with a named owner and a due date for each, even if approximate

## Verification checklist

- [ ] Every timeline event traceable to a real source: a monitoring alert, a ticket, a chat message, a log entry you pulled yourself
- [ ] No log lines, metric values, or error messages present that you did not supply
- [ ] Root cause section matches what your team concluded, not what the AI inferred from the symptoms
- [ ] Impact figures verified against your actual monitoring dashboards or support ticket counts
- [ ] Action items reviewed by the owners named before the document is published
- [ ] Blameless tone confirmed throughout: re-read every sentence for individual finger-pointing
- [ ] Draft shared with the incident commander or engineering lead for accuracy before distribution

## Sample output

Sample output. Generated by AI from sample inputs:

# Postmortem: Database Connection Pool Exhaustion (INC-2026-0412)

**Date:** 2026-06-09
**Duration:** 47 minutes (14:22-15:09 UTC)
**Severity:** SEV-2
**Services affected:** Northwind Order API, Northwind Customer Portal
**Written by:** Platform Engineering team

## Incident Summary

On 2026-06-09, the Northwind Order API became unavailable for 47 minutes due to database connection pool exhaustion on DB-PRIMARY-01. Customer-facing order placement was unavailable. Browsing and authentication were unaffected. The incident was resolved by restarting the API service, which released stale connections, and by deploying a configuration change to reduce the per-instance connection limit.

## Timeline

| Time (UTC) | Event |
|---|---|
| 14:22 | PagerDuty alert: Order API health check failing |
| 14:24 | On-call engineer acknowledged alert |
| 14:31 | Database connection count confirmed at 500/500 (pool exhausted) |
| 14:38 | Decision made to restart API pods to release stale connections |
| 14:44 | First API pod restarted; partial recovery observed |
| 14:55 | All API pods restarted; service recovered |
| 15:09 | Incident declared resolved after 10-minute monitoring window |

## Root Cause

A deployment at 13:55 UTC increased the number of Order API instances from 4 to 8 as part of a planned scale-up for the summer traffic period. The per-instance connection pool limit was not adjusted. The 8 instances collectively opened 500 connections, exhausting the database's configured maximum.

## Contributing Factors

- The deployment runbook did not include a step to recalculate the per-instance pool limit when instance count changes.
- The database connection limit was not included in the pre-deployment checklist.
- No alert existed for database connection count approaching the ceiling.

## Impact

Order placement was unavailable for approximately 47 minutes. Per support data, 312 order attempts failed during the window. No data loss occurred.

## What Went Well

- On-call was paged within 2 minutes of the first failing health check.
- The team correctly identified the root cause within 9 minutes of acknowledging the alert.
- The database connection metric was available in the existing dashboard; detection was fast once the engineer looked.

## Action Items

| Action | Owner | Due |
|---|---|---|
| Add per-instance connection pool recalculation step to scale-up runbook | T. Reyes | 2026-06-16 |
| Create alert for database connections above 80% of configured maximum | D. Park | 2026-06-16 |
| Add database connection ceiling to pre-deployment checklist | Platform Engineering | 2026-06-20 |

## Lessons Learned

Scale-up events need a connection budget review as a required step, not an afterthought. A single alert on database connection headroom would have surfaced this before customers were affected.

## Customization notes

For customer-facing postmortems, strip internal hostnames and tool names and replace them with plain descriptions. If your org tracks a five-whys analysis, add a [FIVE WHYS] input variable and request a five-whys section after Root Cause. Security incidents may require a separate security-specific postmortem format for audit or compliance purposes; check with your security team before using this template for a security incident.

---

From The IT & Cybersecurity Library at vetslearn.com/libraries/it-cyber