AI threat modeling and adversarial testing
Find and block the action an attacker is trying to cause.
Reviewed: 2026-10-03
Source-based teaching example · Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — What you will learn

Find and block the action an attacker is trying to cause.

Visual description:
Find and block the action an attacker is trying to cause.

00:00:06.375 — The idea

Map what needs protection and where trust changes. Test attacks in the supplied sandbox. Verify that forbidden execution stays blocked even if the model’s words change.

Visual description:
Map what needs protection and where trust changes. Test attacks in the supplied sandbox. Verify that forbidden execution stays blocked even if the model’s words change.

00:00:20.708 — One useful term

Trust boundary. A point where information or an action crosses between different levels of permission or trust.

Visual description:
A point where information or an action crosses between different levels of permission or trust.

00:00:30.708 — A worked example

The vulnerable seed follows instructions inside retrieved text and exposes export. The fix treats source text as untrusted, removes export by default and checks permission in application code.

Visual description:
The vulnerable seed follows instructions inside retrieved text and exposes export. The fix treats source text as untrusted, removes export by default and checks permission in application code.

00:00:46.208 — Your turn to think

Is testing refusal wording enough to prove export never ran? Pause and decide before continuing.

Visual description:
Is testing refusal wording enough to prove export never ran?
Pause the video if you need more time.

00:01:03.417 — Why this answer works

No. Assert that forbidden actions produce no execution. Add the failed attack as a repeatable regression case.

Visual description:
No. Assert that forbidden actions produce no execution. Add the failed attack as a repeatable regression case.

00:01:14.208 — Check 1

Check actual effects.

Visual description:
Check actual effects.

00:01:18.875 — Check 2

Enforce permission outside model output.

Visual description:
Enforce permission outside model output.

00:01:24.458 — Check 3

Use only the supplied sandbox targets.

Visual description:
Use only the supplied sandbox targets.

00:01:30.208 — Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

Visual description:
Open the lesson
Try the new case
Check and revise
The full task and its requirements are in the linked lesson.