Vendor evaluation that doesn't get gamed
Choose candidates using critical rules before average scores.
In this lesson · 02 of 7
All lessons in this track →Saved in this browser. View your learning.
Start with the idea
Choose candidates using critical rules before average scores.
Define important failure rules before testing vendors. Keep every failed critical case visible. A candidate that passes the task test can still need operational evidence.
Critical gate: A required rule whose failure prevents an unconditional recommendation.
See the example and check your understanding
A made-up teaching example
A answers 18 of 20 cases correctly but invents both critical deadlines. B answers 17 correctly and declines to guess those deadlines. A fails the predeclared rule.
Ask yourself
Should A win because its average is higher?
Show the explanation
No. B is only a conditional candidate while review effort and operational evidence remain unknown.
Start with these checks
- Apply gates before scores.
- Retain critical failures.
- Show missing lifecycle evidence.
Use the different practice case below and check every requirement in its full instructions.
The video explains the main idea. Use the sources, instructions and checks below to complete the practice.
Captions appear here during playback.
Vendor evaluation that doesn't get gamed
Choose candidates using critical rules before average scores.
Video playback is unavailable. Read the descriptive transcript below, go directly to practice, or open the video file.
Source-based teaching example · Fictional data · Reviewed 2026-10-03. Microsoft Mark · locally synthesized. On a small screen, use fullscreen for diagram details or read the transcript.
Read the descriptive transcript
0:00 · What you will learn
Choose candidates using critical rules before average scores.
What appears on screen
Choose candidates using critical rules before average scores.
0:06 · The idea
Define important failure rules before testing vendors. Keep every failed critical case visible. A candidate that passes the task test can still need operational evidence.
What appears on screen
Define important failure rules before testing vendors. Keep every failed critical case visible. A candidate that passes the task test can still need operational evidence.
0:21 · One useful term
Critical gate. A required rule whose failure prevents an unconditional recommendation.
What appears on screen
A required rule whose failure prevents an unconditional recommendation.
0:29 · A worked example
A answers 18 of 20 cases correctly but invents both critical deadlines. B answers 17 correctly and declines to guess those deadlines. A fails the predeclared rule.
What appears on screen
A answers 18 of 20 cases correctly but invents both critical deadlines. B answers 17 correctly and declines to guess those deadlines. A fails the predeclared rule.
0:44 · Your turn to think
Should A win because its average is higher? Pause and decide before continuing.
What appears on screen
Should A win because its average is higher? Pause the video if you need more time.
1:00 · Why this answer works
No. B is only a conditional candidate while review effort and operational evidence remain unknown.
What appears on screen
No. B is only a conditional candidate while review effort and operational evidence remain unknown.
1:08 · Check 1
Apply gates before scores.
What appears on screen
Apply gates before scores.
1:13 · Check 2
Retain critical failures.
What appears on screen
Retain critical failures.
1:18 · Check 3
Show missing lifecycle evidence.
What appears on screen
Show missing lifecycle evidence.
1:23 · Now try a different case
Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.
What appears on screen
Open the lesson Try the new case Check and revise The full task and its requirements are in the linked lesson.
Watching explains the example. Complete the lesson practice to check what you can do. Explore all visual lessons →
Run a reproducible mission comparison. Build a judgment you can explain and a result another reader can inspect.
Prerequisites: know the difference between generated text and source evidence. For configuration vocabulary, use the AI workflow reference. This is public self-study, not official government training, accreditation, a clearance or a professional credential.
Observable outcomes: Precommit a synthetic mission comparison, score critical failures separately and make a conditional recommendation that records configuration differences.
Part 1 Run a reproducible mission comparison
Predeclare the mission task, non-AI baseline, representative inputs, critical cases, scoring rules and exclusions. Evaluate complete configurations: model version, interface, retrieval corpus, harness tools and permissions, resource budget, retries and human review. A chat comparison, raw API test and agent workflow do different work; document differences instead of attributing them all to model quality.
Use common inputs and blinded scoring where feasible. Keep a holdout set separate from tuning examples; record raw outputs, failures, rater disagreement and exact settings. Critical disclosures or unauthorized actions can block selection even when average quality is high. Measure completion quality, review burden, latency, availability, accessibility and exit cost; public benchmarks and marketing remain supporting evidence.
A real procurement must follow the responsible acquisition officials and current rules. Engineers provide evidence and recommendations. Do not transmit protected sample data or contact references as a classroom drill.
Part 2 Synthetic worked example
Synthetic packet: A answers 18/20 cases correctly but fabricates both critical deadline answers. B answers 17/20 and abstains on both deadlines. The predeclared rule requires correct source-supported critical answers or abstention. A fails that gate. B is a conditional candidate, pending review-time and operational evidence, not automatically a winner.
Guided practice: write the tempting conclusion first, then underline the fact that does not support it. Rewrite the conclusion with that limitation visible. Identify one additional observation that would change the decision.
Next step: carry your artifact into measurable requirements in lesson 03; retain assumptions and unanswered questions.