Software quality, testing, and safe telemetry

What you will be able to do

1. Choose unit, integration, and contract tests.
2. Create CI-quality gates for behavior and controls.
3. Apply an allowlist to telemetry and test redaction.

Prerequisites: dictionaries, loops, functions, exceptions and installed Python or paper tracing. Reuse earlier Builder artifacts only when the task requires them.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

AI behavior does not excuse weak software engineering. Unit tests isolate deterministic logic; contract tests freeze interfaces; integration tests verify component interaction; end-to-end tests cover critical user paths; adversarial tests exercise abuse and failure. Telemetry should begin with an allowlist of operational metadata. Redaction after capture is weaker because sensitive content has already entered the logging system. Record event type, request ID, component, timing, result class, and approved derived measures. Full prompts and responses require a documented purpose, access control, retention period, and deletion route.

Mechanism and implementation

Tests cover deterministic contracts without calling a model: unit for local logic, contract for statuses/schemas, integration for access/routing, and behavioral eval for task quality. Minimize telemetry when constructing it. A redactor after capture can miss nested values, Unicode variants and exception strings.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

def safe_event(event):
    allowed = {"request_id", "status", "latency_ms", "task"}
    return {key: event[key] for key in allowed if key in event}
raw = {"request_id": "r1", "status": "complete", "prompt": "private",
       "token": "synthetic-secret", "nested": {"email": "example.invalid"}}
logged = safe_event(raw)
assert set(logged) == {"request_id", "status"}
assert "synthetic-secret" not in str(logged)
print("allowlist check passed")

Engineering decisions

• Place each failure at the cheapest test layer that can detect it.
• Define CI gates that cannot be waived by the code author alone.
• Select telemetry fields from an allowlist rather than a denylist.

Worked example

A request object contains name, email, token, and task category. The safe logger emits only request ID, task category, status, and latency. Tests seed multiple secret formats and require none to appear in captured output.

Diagnose the failure

Only declared fields appear. If a new field leaks, fix the allowlist entry point and rerun all paths. Capturing full content before redaction fails the intended boundary.

Part 2 Guided practice

Seed secrets in input, nested metadata and exceptions. Capture logs and prove none serialize. Independently add an input field and ensure it remains excluded without a logger edit.

Answer criteria and feedback: Only declared fields appear. If a new field leaks, fix the allowlist entry point and rerun all paths. Capturing full content before redaction fails the intended boundary.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

When a privacy test fails, treat the captured log as sensitive lab evidence, clear the local test output, identify the entry point, and tighten the allowlist. Do not expand the denylist as the primary fix. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://csrc.nist.gov/pubs/sp/800/218/final
• https://www.cisa.gov/securebydesign

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.