Use lessons 01–02 or equivalent skills before this exercise. You can revisit them from the lesson list.
A text editor and supplied fictional decision packets. No purchase, procurement action or operational approval is performed.
Start with the idea
Ask what a vendor’s headline score actually measured.
Request the case selection, scoring rule and configuration. Test failures relevant to your own task. Keep vendor-reported results distinct from independently observed evidence.
Measurement scope: The tasks, cases and conditions a reported result actually covers.
See the example and check your understanding
A made-up teaching example
The vendor claims 98 percent accuracy on one hundred selected questions judged by its own staff. The test omits missing-answer and permission cases, and the configuration is unspecified.
Ask yourself
Does that score establish performance on your workflow?
Show the explanation
No. Ask for the method and relevant negative-case results. Do not endorse savings that were not measured.
Start with these checks
- Check case and judge selection.
- Include missing-evidence and permission failures.
- Separate reported and observed results.
Use the different practice case below and check every requirement in its full instructions.
Full learning objectives
- Define representative tasks and failure cases.
- Ask for configuration and result evidence.
- Separate certifications from demonstrated task performance.
Captions appear here during playback.
Evaluate vendor claims with task-specific evidence
Ask what a vendor’s headline score actually measured.
Video playback is unavailable. Read the descriptive transcript below, go directly to practice, or open the video file.
Source-based teaching example · Fictional data · Reviewed 2026-10-03. Microsoft Mark · locally synthesized. On a small screen, use fullscreen for diagram details or read the transcript.
Read the descriptive transcript
0:00 · What you will learn
Ask what a vendor’s headline score actually measured.
What appears on screen
Ask what a vendor’s headline score actually measured.
0:06 · The idea
Request the case selection, scoring rule and configuration. Test failures relevant to your own task. Keep vendor-reported results distinct from independently observed evidence.
What appears on screen
Request the case selection, scoring rule and configuration. Test failures relevant to your own task. Keep vendor-reported results distinct from independently observed evidence.
0:21 · One useful term
Measurement scope. The tasks, cases and conditions a reported result actually covers.
What appears on screen
The tasks, cases and conditions a reported result actually covers.
0:30 · A worked example
The vendor claims 98 percent accuracy on one hundred selected questions judged by its own staff. The test omits missing-answer and permission cases, and the configuration is unspecified.
What appears on screen
The vendor claims 98 percent accuracy on one hundred selected questions judged by its own staff. The test omits missing-answer and permission cases, and the configuration is unspecified.
0:46 · Your turn to think
Does that score establish performance on your workflow? Pause and decide before continuing.
What appears on screen
Does that score establish performance on your workflow? Pause the video if you need more time.
1:02 · Why this answer works
No. Ask for the method and relevant negative-case results. Do not endorse savings that were not measured.
What appears on screen
No. Ask for the method and relevant negative-case results. Do not endorse savings that were not measured.
1:13 · Check 1
Check case and judge selection.
What appears on screen
Check case and judge selection.
1:17 · Check 2
Include missing-evidence and permission failures.
What appears on screen
Include missing-evidence and permission failures.
1:23 · Check 3
Separate reported and observed results.
What appears on screen
Separate reported and observed results.
1:29 · Now try a different case
Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.
What appears on screen
Open the lesson Try the new case Check and revise The full task and its requirements are in the linked lesson.
Watching explains the example. Complete the lesson practice to check what you can do. Explore all visual lessons →
The method
Translate claims such as “accurate” or “secure” into questions about a specific workflow. Ask what was tested, which cases failed, who scored results, how the model and harness were configured, and what the result excludes. Use your own synthetic cases when appropriate; a rehearsed vendor demonstration may not represent your work.
Compare options under equivalent criteria, including human review, operational limits, recovery and maintenance. An agreement or credential may be relevant evidence for a particular requirement, but it does not prove factual accuracy or every compliance condition. This lesson creates an evaluation packet; it is not legal advice or an actual procurement eligibility assessment.
Choose your model, harness and offline route
Text-only assistant
Use supplied text. Ask for a plan or draft; do not claim browsing, code execution or file access that was unavailable.
Tool-enabled application or harness
Record which tools actually run, allowed files and actions, approval boundaries and observable results. Tools are optional unless the task needs them.
Local or cloud deployment
Inspect inference, retrieval, connectors, logs and storage separately. Use only authorized environments for real data; training fixtures are fictional.
Higher-capability or frontier model
Evaluate the exact task and configuration. Capability does not supply missing evidence or grant permission. No provider ranking or guarantee is implied.
Offline alternative
Use the supplied inputs and inspect the worked example manually. Builder exercises use local deterministic Python. No paid account or live model call is required.
Record the exact product/model label when available, tools actually used, inputs, permitted actions, observed output and checks. Live accounts or APIs may have external costs; they are optional and are not used by this page.
Read the general workflow guide →Worked example
Supplied input
Fictional vendor claim: 98% accurate. Evidence: 100 selected questions, 98 judged acceptable by vendor staff; no absent-answer or permission cases; configuration unspecified.
A checked result
Record the result as vendor-reported on selected cases. Request scoring criteria, case selection, configuration and negative-case results. Do not generalize to your tasks.
Why this works
A number without its measurement method cannot answer the acceptance question.
Try a different case
Independent vendor claim: “cuts work in half.” Demo timed generation only on five easy tasks; review and correction excluded. Write a response requesting evidence and define six acceptance cases.
Download this task and checklist →Interactive practice / Fictional fixture
Challenge an incomplete vendor metric
Use the supplied independent case above. Check the bounded decisions, then review your written artifact against the full acceptance criteria.
Everything runs in this browser. No AI model is called and no work is uploaded. Bounded checks cover specific facts and rules. Your writing is not graded: you review it yourself. Practice never marks a lesson complete automatically.
Include absent evidence, restricted access and malformed input. Do not endorse the savings claim.
Try the task, run the checks, then revise. Checks are teaching feedback, not an instructor grade.
Observed browser simulation
This trace comes from the bounded local checker. It is not a Python run, model result or production tool execution.
Self-review
Locked. Write every written part, then run the checks. A few words is not an attempt.
This is self-review, not grading. Read our answers, then tick what your own work does. Ticks are optional. They stay on this page and are not saved or sent anywhere. A different answer can be just as good if it meets the lesson criteria.
Your vendor response and six acceptance cases
A strong answer
Thanks for the demo. Before we can assess the savings claim we need: how the five tasks were chosen; time for preparation, review and correction, not generation alone; results on tasks that failed; the exact product configuration used. We would like to run six cases of our own: 1) a routine task with full sources; 2) a task where the answer is not in the sources; 3) a request for restricted material; 4) malformed input; 5) a task with conflicting sources; 6) a long task that needs correction. Pass means a usable output with total effort recorded. We are not accepting the savings figure at this point.
A flawed answer
The demo was impressive and cutting work in half would be valuable. Please send pricing. Acceptance cases: six typical tasks completed quickly.
What is wrong with it: It accepts a claim built on generation time for five easy tasks. All six cases are easy ones, so nothing tests missing evidence, restricted access or bad input. “Quickly” is not a pass standard.
Keep or restore your work
Drafts are not saved automatically. Save only fictional practice data. A browser draft may be lost if storage is cleared; download a copy to keep it.
Your steps
- Ask for total usable-task effort and quality results.
- Include absent evidence, restricted access and a malformed input.
- Record unresolved evidence rather than choosing a winner prematurely.
Check your work
- No unsupported savings endorsement.
- Cases reflect the intended workflow.
- Questions distinguish reported results from independently observed evidence.
After your attempt: review notes and a common mistake
Request preparation, review, correction and failure handling alongside generation time. A passing demo on five selected tasks cannot establish all-work savings.
Different wording or a different justified decision can pass. Check your work against every requirement; these notes are not an automated grade.
Explain your approach
Which vendor answer would materially change your evaluation?
Keep your artifact, one correction or decision, its evidence and one remaining limitation. Nothing on this page sends or uploads your work.
Check your understanding
Choose an answer to see feedback. This is a knowledge check, not instructor assessment.
Review your independent practice
Complete the supplied task before recording progress. Use the acceptance criteria and reveal the review notes after your attempt. Keep your artifact yourself; nothing is uploaded.
Not yet self-reviewed. Completion stays in this browser and does not issue a badge.
Optional deeper study
Explore the related general-course lesson → This opens the existing canonical lesson; it is optional depth, not another copy of this focused task.
Revision 2026-10-03. Fictional examples, not observed provider results. Knowledge checks and completion records are self-study aids.