Machine learning foundations for system builders
Check whether a model’s confidence matches its observed results.
Reviewed: 2026-10-03
Source-based teaching example · Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — What you will learn

Check whether a model’s confidence matches its observed results.

Visual description:
Check whether a model’s confidence matches its observed results.

00:00:06.750 — The idea

Compare predictions with known labels. Then examine whether stated confidence is justified across relevant kinds of input. A small arithmetic example cannot establish population-wide performance.

Visual description:
Compare predictions with known labels. Then examine whether stated confidence is justified across relevant kinds of input. A small arithmetic example cannot establish population-wide performance.

00:00:22.792 — One useful term

Calibration. How closely predicted probabilities match observed outcomes under the evaluated conditions.

Visual description:
How closely predicted probabilities match observed outcomes under the evaluated conditions.

00:00:32.917 — A worked example

The urgency classifier looks accurate but becomes overconfident when wording changes. The learner records that failure and considers when the system should decline to decide.

Visual description:
The urgency classifier looks accurate but becomes overconfident when wording changes. The learner records that failure and considers when the system should decline to decide.

00:00:46.333 — Your turn to think

Does strong accuracy on familiar wording prove reliable confidence on new wording? Pause and decide before continuing.

Visual description:
Does strong accuracy on familiar wording prove reliable confidence on new wording?
Pause the video if you need more time.

00:01:05.000 — Why this answer works

No. Test the changed conditions. Check that labels align with predictions before blaming training or retraining.

Visual description:
No. Test the changed conditions. Check that labels align with predictions before blaming training or retraining.

00:01:15.583 — Check 1

Check label and prediction alignment.

Visual description:
Check label and prediction alignment.

00:01:20.750 — Check 2

Inspect changed-input cases.

Visual description:
Inspect changed-input cases.

00:01:25.833 — Check 3

Limit conclusions to the tested evidence.

Visual description:
Limit conclusions to the tested evidence.

00:01:31.292 — Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

Visual description:
Open the lesson
Try the new case
Check and revise
The full task and its requirements are in the linked lesson.