Evaluate vendor claims with task-specific evidence
Ask what a vendor’s headline score actually measured.
Reviewed: 2026-10-03
Source-based teaching example · Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — What you will learn

Ask what a vendor’s headline score actually measured.

Visual description:
Ask what a vendor’s headline score actually measured.

00:00:06.042 — The idea

Request the case selection, scoring rule and configuration. Test failures relevant to your own task. Keep vendor-reported results distinct from independently observed evidence.

Visual description:
Request the case selection, scoring rule and configuration. Test failures relevant to your own task. Keep vendor-reported results distinct from independently observed evidence.

00:00:21.083 — One useful term

Measurement scope. The tasks, cases and conditions a reported result actually covers.

Visual description:
The tasks, cases and conditions a reported result actually covers.

00:00:30.667 — A worked example

The vendor claims 98 percent accuracy on one hundred selected questions judged by its own staff. The test omits missing-answer and permission cases, and the configuration is unspecified.

Visual description:
The vendor claims 98 percent accuracy on one hundred selected questions judged by its own staff. The test omits missing-answer and permission cases, and the configuration is unspecified.

00:00:46.000 — Your turn to think

Does that score establish performance on your workflow? Pause and decide before continuing.

Visual description:
Does that score establish performance on your workflow?
Pause the video if you need more time.

00:01:02.833 — Why this answer works

No. Ask for the method and relevant negative-case results. Do not endorse savings that were not measured.

Visual description:
No. Ask for the method and relevant negative-case results. Do not endorse savings that were not measured.

00:01:13.042 — Check 1

Check case and judge selection.

Visual description:
Check case and judge selection.

00:01:17.875 — Check 2

Include missing-evidence and permission failures.

Visual description:
Include missing-evidence and permission failures.

00:01:23.750 — Check 3

Separate reported and observed results.

Visual description:
Separate reported and observed results.

00:01:29.250 — Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

Visual description:
Open the lesson
Try the new case
Check and revise
The full task and its requirements are in the linked lesson.