Observability, safe logging, and incident response

What you will be able to do

1. Define signals across application, model, retrieval, and tools.
2. Implement privacy-aware traces.
3. Detect, contain, recover, communicate, and learn.

Prerequisites: Builder-level interface validation, Python, testing, metadata logging and state-transition tracing. Hardware benchmarks and model accounts are not required.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

Observability connects symptoms to causes using metrics, logs, traces, and audit records. Signals should cover service health, model behavior, retrieval evidence, tool outcomes, safety events, and mission impact. Full content is exceptional; use identifiers, derived measures, sampling, and controlled diagnostic capture. An incident process protects people and systems, preserves evidence, assigns decision authority, contains spread, restores a known-good state, communicates to required parties, and converts the failure into regression evidence. A restart without cause and test evidence is not recovery.

Mechanism and implementation

Signals should diagnose readiness, queues, output states, support, denials and effects. Content capture is exceptional and scoped; incidents do not justify indefinite retention. Named owners choose containment and recovery, with artifact/data versions and post-restore checks.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

events = [{"id": "e1", "kind": "denied", "version": "v3"},
          {"id": "e2", "kind": "critical_disclosure", "version": "v3"}]
severity = "critical" if any(e["kind"] == "critical_disclosure" for e in events) else "review"
route_enabled = severity != "critical"
assert not route_enabled
preserved = [{k: e[k] for k in ("id", "kind", "version")} for e in events]
assert all("content" not in e for e in preserved)
print("route contained; metadata preserved")

Engineering decisions

• Define severity and escalation triggers.
• Minimize telemetry while preserving diagnostic value.
• Choose containment before optimization.

Worked example

An incident inject raises cross-tenant denial events and a groundedness drop after release v3. The operator freezes promotion, disables the affected path, preserves metadata traces, rolls back, validates recovery, and opens a regression action.

Diagnose the failure

Contain the route first. A restart alone is not recovery: repeat the original negative-access and normal cases. Preserve original timing and never rewrite incident evidence.

Part 2 Guided practice

Distinguish healthy denials from disclosure. Trace detection, containment, evidence, rollback and verification. Independently set record-specific retention, access/deletion and any legitimate hold owner.

Answer criteria and feedback: Contain the route first. A restart alone is not recovery: repeat the original negative-access and normal cases. Preserve original timing and never rewrite incident evidence.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

Use the reset manifest to restore the simulator. Preserve the original incident events; never edit them to make the response appear faster. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://csrc.nist.gov/pubs/sp/800/61/r3/final
• https://airc.nist.gov/airmf-resources/airmf/

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.