Watch the explanation. Practice the skill.
Machine learning foundations for system builders
Check whether a model’s confidence matches its observed results.
Captions appear here during playback.
Machine learning foundations for system builders
Check whether a model’s confidence matches its observed results.
Video playback is unavailable. Read the descriptive transcript below, go directly to practice, or open the video file.
Source-based teaching example · Fictional data · Reviewed 2026-10-03. Microsoft Mark · locally synthesized. On a small screen, use fullscreen for diagram details or read the transcript.
Read the descriptive transcript
0:00 · What you will learn
Check whether a model’s confidence matches its observed results.
What appears on screen
Check whether a model’s confidence matches its observed results.
0:06 · The idea
Compare predictions with known labels. Then examine whether stated confidence is justified across relevant kinds of input. A small arithmetic example cannot establish population-wide performance.
What appears on screen
Compare predictions with known labels. Then examine whether stated confidence is justified across relevant kinds of input. A small arithmetic example cannot establish population-wide performance.
0:22 · One useful term
Calibration. How closely predicted probabilities match observed outcomes under the evaluated conditions.
What appears on screen
How closely predicted probabilities match observed outcomes under the evaluated conditions.
0:32 · A worked example
The urgency classifier looks accurate but becomes overconfident when wording changes. The learner records that failure and considers when the system should decline to decide.
What appears on screen
The urgency classifier looks accurate but becomes overconfident when wording changes. The learner records that failure and considers when the system should decline to decide.
0:46 · Your turn to think
Does strong accuracy on familiar wording prove reliable confidence on new wording? Pause and decide before continuing.
What appears on screen
Does strong accuracy on familiar wording prove reliable confidence on new wording? Pause the video if you need more time.
1:05 · Why this answer works
No. Test the changed conditions. Check that labels align with predictions before blaming training or retraining.
What appears on screen
No. Test the changed conditions. Check that labels align with predictions before blaming training or retraining.
1:15 · Check 1
Check label and prediction alignment.
What appears on screen
Check label and prediction alignment.
1:20 · Check 2
Inspect changed-input cases.
What appears on screen
Inspect changed-input cases.
1:25 · Check 3
Limit conclusions to the tested evidence.
What appears on screen
Limit conclusions to the tested evidence.
1:31 · Now try a different case
Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.
What appears on screen
Open the lesson Try the new case Check and revise The full task and its requirements are in the linked lesson.
Watching explains the example. Complete the lesson practice to check what you can do. Explore all visual lessons →