Monitoring, observability, and incident response

Lesson preparation and outcomes

Prerequisites: Allowlist logging, version records and failure classes.

By the end, you can: Connect safe telemetry, SLOs and incident response to actual consequences.

Worked scenario and practice feedback

A denial spike could show successful containment or a legitimate-user outage; it is not automatically a breach. A verified cross-tenant disclosure is critical: contain the route, preserve minimized evidence, roll back and rerun negative access and normal behavior.

Guided and independent practice: Build a synthetic incident timeline; independently specify capture purpose, retention/deletion, hold owner and post-recovery tests.

Answer criteria: No default full-content logging, cost-only sampling rule or indefinite incident retention; original timing preserved; restoration requires verification. If the local pack or interpreter is unavailable, trace the synthetic input, expected output and failure guard on paper; label execution unverified. Required practice uses no paid API calls, accounts, purchase, operational data or actual deployment.

Tier 02 Lesson 08 taught eval discipline in development. This lesson takes it to production, where the model talks to real users, the load varies, model versions update, and the eval needs to keep running after you have moved on. Silent drift is one risk; monitor service behavior and task quality against declared requirements.

Part 1 The three pillars, applied to AI.

Traditional observability rests on three pillars: metrics, logs, and traces. AI systems need all three plus a fourth: evals in production.

• Metrics. Numerical time-series. Request rate, latency percentiles, token throughput, error rate, GPU utilization. Dashboards live here.
• Logs. Structured event records. Allowlisted events: request ID, task/template version, model revision, result class, approved usage/timing fields and control outcomes. Select storage from access, retention, durability and diagnostic needs; full content is exceptional.
• Traces. Per-request span trees showing where time was spent. Critical for multi-step systems: RAG with retrieval + rerank + generation, agents with tool calls. OpenTelemetry is the open standard.
• Production evals. Evaluate approved production samples or synthetic probes under data-purpose and access rules. Select coverage from critical cases, privacy, representativeness and diagnostic value, then resource budget. Do not capture unauthorized content merely to grade it.

Choose the first signal from the failure you need to detect. A readiness metric may reveal an outage; a bounded trace may locate a timeout; a claim-support eval may reveal wrong answers. More content capture is not inherently better diagnosis.

Part 2 SLOs and SLIs for AI systems.

Pick your service level indicators (SLIs) before the system goes live; set service level objectives (SLOs) based on stakeholder needs.

Common SLIs for AI systems:

• Availability. Percentage of requests that returned a response (not an error or timeout).
• First-token latency (p95). Time from request to first token of the response.
• End-to-end latency (p99). Time to full response.
• Eval pass rate. Percentage of sampled requests that score above the eval threshold.
• Resource cost per completed task. Include applicable inference charges (input, output, cached input and other billed usage), tools, retries, storage/compute and allocated operating costs under declared assumptions. Keep unknown rates unknown; token charges alone are not the full task cost. Record human review/correction effort separately or include it explicitly in a total-cost measure.

Constructed SLO example for discussion, not a default or measured system:

• Availability ≥ 99.5%
• First-token latency p95 ≤ 800ms
• End-to-end latency p99 ≤ 4s
• Eval pass rate ≥ 87% (matching the contract bar)
• Resource cost per completed task ≤ 2 resource units under declared included-cost and allocation assumptions (invented threshold)

Three rules that pay off:

• SLOs are agreed with the responsible service or mission owner. Record the affected stakeholders, trade-offs and decision authority rather than assuming engineering alone sets acceptance.
• SLOs come with error budgets. For the request-based 99.5% definition above, the error budget is 0.005 times eligible requests in the measurement window. Define which requests count and which failure states consume budget. A time-based availability SLI instead uses unavailable time divided by eligible time; do not mix the denominators. Predeclare the response to budget exhaustion.
• SLIs are continuously measured. Not “we’ll check on Friday.” Real-time dashboards your team watches.

Part 3 What to log, and what not to.

Construct routine events from an approved allowlist:

• Request ID (trace this everywhere)
• Timestamp
• A minimized correlation identifier only when permitted; do not assume a user ID is needed
• Model name and version
• Prompt/template and result version identifiers, not content by default
• Result class and approved evidence/control identifiers
• Token counts (input, output)
• Latency
• Cost
• Eval score (if available)
• Outcome signal (did the user accept the response, edit it, reject it)

Full prompt/response capture is exceptional and requires a prior approved diagnostic purpose, minimized scope, access controls, retention and deletion route. Check every telemetry exporter and exception path before capture. Routine event logging never implies routine content logging.

What requires care:

• Personally identifiable information (PII). Avoid collecting PII in routine telemetry. Exceptional content diagnostics need documented purpose, minimum fields, access/retention/deletion rules and approved capture before storage.
• Classified or CUI content. An approved data boundary must include logging, telemetry, backups and support paths. Matching a classification label alone does not grant authorization; use synthetic data for this lab.
• Sensitive operational context. Same as cleared rules from Annex A: what would happen if these logs leaked?

Three patterns that pay off:

• Structured JSON logging. One JSON object per line. Easy to ship, easy to query.
• Sampling at the right rate. Choose sampling and event coverage from diagnostic value, security/audit obligations and privacy constraints. High event coverage never implies full content capture.
• Retention by record type. Set record-specific retention under applicable policy and legitimate preservation requirements. Incidents do not create an indefinite-retention default; define hold owner, review and deletion route.

Part 4 Drift detection: the quiet failure mode.

Silent quality drift can arise when:

• The base model updated and now performs differently on your task.
• The user behavior changed and the prompts coming in don’t match what the system was tuned for.
• The data the system retrieves from has changed.
• The world changed (training data is stale; current events have moved past the cutoff).

A change can be gradual or abrupt. Detect it using a fixed synthetic/regression suite plus permitted representative observations; a shifted distribution is a diagnostic trigger rather than proof of a quality drop.

Three patterns that detect drift:

• Continuous eval sampling. Run a fixed regression/probe set on a declared cadence and separately evaluate permitted traffic samples. Track case IDs, slice scores and sampling changes; alarm on predeclared degradation and critical failures.
• Distributional monitoring. Track statistics of the incoming prompts (lengths, vocabulary, language detection) and outputs (lengths, refusal rates, structured-output validity). Alarm on shifts.
• User outcome signals. Track whether users accept, edit, or reject responses. Edit rate can signal task or user changes; validate the interpretation against sampled task evidence.

Two pitfalls:

• Don’t run eval only when you suspect a problem. By then the problem has compounded.
• Don’t trust user satisfaction as the sole signal. Users adapt to degradation; you’ll miss a real quality drop until it’s bad enough to complain about.

Part 5 The AI incident response playbook.

Most engineering teams have an incident response process for web services. AI systems need an adapted version.

Severity levels:

• SEV-1. System down. Returns errors to most or all users. Or: producing dangerous, sensitive, or wrong-in-public-ways outputs.
• SEV-2. Major degradation. Latency, accuracy, or quality has dropped meaningfully. Most users affected but system still operational.
• SEV-3. Minor degradation. Subset of users or use cases affected. Eval has shifted within thresholds.

On-call response steps:

1. Acknowledge under the agreed response target; the scenario may use 5/30-minute examples, but actual requirements are consequence- and organization-specific.
2. Contain and stabilize. Disable an affected disclosure/action path before further exposure; pause work and route only to a verified safe alternative.
3. Preserve and coordinate. Preserve minimized evidence, name the decision owner and draft required notifications under the actual incident policy. The classroom does not send messages.
4. Diagnose. Pull logs, traces, eval results. Identify the root cause.
5. Resolve. Deploy the fix. Verify against the eval. Restore traffic gradually.
6. Write the postmortem under the organization’s defined completion target. No blame; full timeline; specific follow-up actions.

Three AI-specific incident patterns to expect:

• Model provider silently updated a model version. You get a different version than you measured. Your contract should have prevented this; the runbook should detect it.
• Prompt injection or jailbreak in the wild. A user found a way to make the system produce content it should not. Lesson 06 covers prevention; the runbook should disable the affected path immediately.
• Cost runaway. Usage doubled overnight; tokens billed unexpectedly. The runbook needs cost alarms and a circuit breaker.

Part 6 Postmortems that improve the system.

Postmortems are how the team learns. Done well, every incident makes the system better. Done poorly, the same incidents recur.

A working postmortem structure:

1. Summary. One paragraph. What happened, when, who was affected.
2. Timeline. Specific times, specific actions. What we observed and when we observed it.
3. Root cause. The actual cause, not the proximate trigger.
4. What went well. Detection time, response coordination, communication.
5. What didn’t. Honest. Specific. No blame.
6. Action items. Concrete. Owned by named individuals. With due dates. Tracked to completion.

Two rules that make postmortems land:

• Blameless. Engineers who caused the incident should be in the room helping write the postmortem. Punishment teaches concealment.
• Action items are real. A postmortem with five action items and no follow-through teaches the team that postmortems are theater. Track action items; close them; reference them next time.

Interface/reference limits

Use the workflow reference to distinguish model capability, deployment and harness controls. Provider snippets in this lesson are illustrative, have not been executed against an actual provider in this revision, and require current SDK/model capability verification before separately authorized use.