Evaluation engineering and statistical evidence
Compare systems without averaging away serious failures.
Reviewed: 2026-10-03
Source-based teaching example · Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — What you will learn

Compare systems without averaging away serious failures.

Visual description:
Compare systems without averaging away serious failures.

00:00:06.500 — The idea

Compare the same cases using aligned IDs. Report important groups separately. A statistical interval does not settle permission questions or every source of uncertainty.

Visual description:
Compare the same cases using aligned IDs. Report important groups separately. A statistical interval does not settle permission questions or every source of uncertainty.

00:00:20.458 — One useful term

Paired comparison. Testing two systems on corresponding versions of the same cases.

Visual description:
Testing two systems on corresponding versions of the same cases.

00:00:29.167 — A worked example

In the larger made-up comparison, routine cases improve but two critical disclosures appear. The release decision is still no-go. A positive overall difference can hide a serious new failure.

Visual description:
In the larger made-up comparison, routine cases improve but two critical disclosures appear. The release decision is still no-go. A positive overall difference can hide a serious new failure.

00:00:44.375 — Your turn to think

Can better average performance cancel a critical disclosure? Pause and decide before continuing.

Visual description:
Can better average performance cancel a critical disclosure?
Pause the video if you need more time.

00:01:01.458 — Why this answer works

No. Apply the critical rule separately. If case IDs do not align, stop the comparison before scoring.

Visual description:
No. Apply the critical rule separately. If case IDs do not align, stop the comparison before scoring.

00:01:12.167 — Check 1

Align case IDs before calculating.

Visual description:
Align case IDs before calculating.

00:01:17.625 — Check 2

Report critical failures separately.

Visual description:
Report critical failures separately.

00:01:22.875 — Check 3

State the sampling assumptions and limits.

Visual description:
State the sampling assumptions and limits.

00:01:28.417 — Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

Visual description:
Open the lesson
Try the new case
Check and revise
The full task and its requirements are in the linked lesson.