Evaluation and observability
Test whether an AI result meets the task, not just whether it contains a keyword.
Reviewed: 2026-10-03
Source-based teaching example · Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — What you will learn

Test whether an AI result meets the task, not just whether it contains a keyword.

Visual description:
Test whether an AI result meets the task, not just whether it contains a keyword.

00:00:08.125 — The idea

Write checks tied to what a good result means. Include misleading wording and important failures. Review unclear judgments and test the logging rules separately.

Visual description:
Write checks tied to what a good result means. Include misleading wording and important failures. Review unclear judgments and test the logging rules separately.

00:00:21.250 — One useful term

Evaluation. Testing results against defined expectations using suitable cases.

Visual description:
Testing results against defined expectations using suitable cases.

00:00:30.167 — A worked example

A keyword check finds “logistics” inside “no logistics experience” and passes it. The word is present, but the meaning is the opposite of the desired claim.

Visual description:
A keyword check finds “logistics” inside “no logistics experience” and passes it. The word is present, but the meaning is the opposite of the desired claim.

00:00:43.208 — Your turn to think

Does the right keyword prove that the required experience is stated? Pause and decide before continuing.

Visual description:
Does the right keyword prove that the required experience is stated?
Pause the video if you need more time.

00:01:00.667 — Why this answer works

No. Check meaning and evidence. Treat a keyword test as an early screen, not a complete quality judgment.

Visual description:
No. Check meaning and evidence. Treat a keyword test as an early screen, not a complete quality judgment.

00:01:11.250 — Check 1

Keep stable test-case IDs.

Visual description:
Keep stable test-case IDs.

00:01:16.333 — Check 2

Report important failures separately.

Visual description:
Report important failures separately.

00:01:21.667 — Check 3

Check that logs contain only allowed fields.

Visual description:
Check that logs contain only allowed fields.

00:01:27.375 — Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

Visual description:
Open the lesson
Try the new case
Check and revise
The full task and its requirements are in the linked lesson.