Watch the explanation. Practice the skill.
Evaluate vendor claims with task-specific evidence
Ask what a vendor’s headline score actually measured.
Captions appear here during playback.
Evaluate vendor claims with task-specific evidence
Ask what a vendor’s headline score actually measured.
Video playback is unavailable. Read the descriptive transcript below, go directly to practice, or open the video file.
Source-based teaching example · Fictional data · Reviewed 2026-10-03. Microsoft Mark · locally synthesized. On a small screen, use fullscreen for diagram details or read the transcript.
Read the descriptive transcript
0:00 · What you will learn
Ask what a vendor’s headline score actually measured.
What appears on screen
Ask what a vendor’s headline score actually measured.
0:06 · The idea
Request the case selection, scoring rule and configuration. Test failures relevant to your own task. Keep vendor-reported results distinct from independently observed evidence.
What appears on screen
Request the case selection, scoring rule and configuration. Test failures relevant to your own task. Keep vendor-reported results distinct from independently observed evidence.
0:21 · One useful term
Measurement scope. The tasks, cases and conditions a reported result actually covers.
What appears on screen
The tasks, cases and conditions a reported result actually covers.
0:30 · A worked example
The vendor claims 98 percent accuracy on one hundred selected questions judged by its own staff. The test omits missing-answer and permission cases, and the configuration is unspecified.
What appears on screen
The vendor claims 98 percent accuracy on one hundred selected questions judged by its own staff. The test omits missing-answer and permission cases, and the configuration is unspecified.
0:46 · Your turn to think
Does that score establish performance on your workflow? Pause and decide before continuing.
What appears on screen
Does that score establish performance on your workflow? Pause the video if you need more time.
1:02 · Why this answer works
No. Ask for the method and relevant negative-case results. Do not endorse savings that were not measured.
What appears on screen
No. Ask for the method and relevant negative-case results. Do not endorse savings that were not measured.
1:13 · Check 1
Check case and judge selection.
What appears on screen
Check case and judge selection.
1:17 · Check 2
Include missing-evidence and permission failures.
What appears on screen
Include missing-evidence and permission failures.
1:23 · Check 3
Separate reported and observed results.
What appears on screen
Separate reported and observed results.
1:29 · Now try a different case
Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.
What appears on screen
Open the lesson Try the new case Check and revise The full task and its requirements are in the linked lesson.
Watching explains the example. Complete the lesson practice to check what you can do. Explore all visual lessons →