ML and deep learning: enough to be dangerous
Judge a classifier by its errors as well as its accuracy.
Reviewed: 2026-10-03
Source-based teaching example · Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — What you will learn

Judge a classifier by its errors as well as its accuracy.

Visual description:
Judge a classifier by its errors as well as its accuracy.

00:00:06.375 — The idea

Count correct and incorrect predictions against known labels. Different mistakes can have different consequences. Training progress alone does not decide whether a system is suitable for use.

Visual description:
Count correct and incorrect predictions against known labels. Different mistakes can have different consequences. Training progress alone does not decide whether a system is suitable for use.

00:00:21.500 — One useful term

Classifier. A model or rule that assigns an input to a category.

Visual description:
A model or rule that assigns an input to a category.

00:00:29.042 — A worked example

The toy example has four cases: two correct predictions and two mistakes. Accuracy is one half. One missed positive and one false alarm can still matter differently to the task.

Visual description:
The toy example has four cases: two correct predictions and two mistakes. Accuracy is one half. One missed positive and one false alarm can still matter differently to the task.

00:00:44.208 — Your turn to think

Does the same accuracy score mean two systems have the same risk? Pause and decide before continuing.

Visual description:
Does the same accuracy score mean two systems have the same risk?
Pause the video if you need more time.

00:01:01.750 — Why this answer works

No. Inspect which errors occurred and what each would cause. Keep test data separate from data used to fit the model.

Visual description:
No. Inspect which errors occurred and what each would cause. Keep test data separate from data used to fit the model.

00:01:12.375 — Check 1

Check counts and the totals you divide by.

Visual description:
Check counts and the totals you divide by.

00:01:17.750 — Check 2

Prevent information leaking from test data into training.

Visual description:
Prevent information leaking from test data into training.

00:01:24.083 — Check 3

Judge error costs for the actual task.

Visual description:
Judge error costs for the actual task.

00:01:29.708 — Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

Visual description:
Open the lesson
Try the new case
Check and revise
The full task and its requirements are in the linked lesson.