AI threat modeling and adversarial testing

What you will be able to do

1. Map assets, actors, trust boundaries, and abuse paths.
2. Test injection, disclosure, poisoning, denial, and excessive agency.
3. Verify mitigations with regression evidence.

Prerequisites: Builder-level interface validation, Python, testing, metadata logging and state-transition tracing. Hardware benchmarks and model accounts are not required.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

AI threat modeling covers the whole system: users, identities, prompts, retrieved content, model, tools, data stores, telemetry, dependencies, operators, and supply chain. Begin with assets and unacceptable outcomes, then draw trust boundaries and entry points. A prompt injection is one attack path, not the threat model. Record preconditions, exploit path, affected asset, impact, existing controls, evidence, and residual risk. Test only in the authorized course sandbox. A mitigation is not complete until a regression test demonstrates containment.

Mechanism and implementation

Injection converts untrusted content into instructions or authority. Treat retrieved text and observations as data; authorize effects in an authenticated application boundary. Test multiple placements, including caches/logs/exports. A model refusal alone does not demonstrate containment.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

allowed = {"read_public"}
def dispatch(action, source):
    return "allowed" if action in allowed else "denied"
assert dispatch("export_private", "trusted-looking-document") == "denied"
assert dispatch("read_public", "user") == "allowed"
assert dispatch("grant_admin", "tool_observation") == "denied"
print("authority escalation blocked")

Engineering decisions

• Prioritize by plausible path and consequence.
• Distinguish prevention, detection, containment, and recovery controls.
• Stop testing at the authorized boundary.

Worked example

The vulnerable seed application trusts retrieved instructions and exposes an export tool. The red team demonstrates an indirect injection; the fix treats retrieval as untrusted, removes the tool by default, validates policy server-side, and adds a regression case.

Diagnose the failure

Forbidden dispatch stays denied regardless of source. Add zero-execution assertions if only refusal wording was tested. Use only the supplied synthetic sandbox.

Part 2 Guided practice

Draw a document-to-model-to-tool attack crossing. Seed a synthetic instruction and test the executor even when the mock model follows it. Independently test an egress target outside the allowlist.

Answer criteria and feedback: Forbidden dispatch stays denied regardless of source. Add zero-execution assertions if only refusal wording was tested. Use only the supplied synthetic sandbox.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

Reproduce one path in the local simulator. If a scenario would touch a real system, stop and translate it into a synthetic equivalent. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://genai.owasp.org/
• https://airc.nist.gov/Docs/Generative_AI_Profile.pdf

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.