Production architecture and inference serving
Test service capacity under bursts, not just average demand.
Reviewed: 2026-10-03
Source-based teaching example · Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — What you will learn

Test service capacity under bursts, not just average demand.

Visual description:
Test service capacity under bursts, not just average demand.

00:00:07.292 — The idea

A system can handle typical traffic and fail when requests arrive together. Check service rate, waiting and fallback behavior. Measure the test setup’s own limits too.

Visual description:
A system can handle typical traffic and fail when requests arrive together. Check service rate, waiting and fallback behavior. Measure the test setup’s own limits too.

00:00:21.167 — One useful term

p95 latency. The time at or below which 95 percent of the measured requests completed.

Visual description:
The time at or below which 95 percent of the measured requests completed.

00:00:30.833 — A worked example

Two simulated designs have different service rates and queue limits. One meets a median target but fails the p95 target during bursts. The decision records the alternative and an offline fallback.

Visual description:
Two simulated designs have different service rates and queue limits. One meets a median target but fails the p95 target during bursts. The decision records the alternative and an offline fallback.

00:00:47.042 — Your turn to think

Does a queue limit alone prove the p95 target? Pause and decide before continuing.

Visual description:
Does a queue limit alone prove the p95 target?
Pause the video if you need more time.

00:01:04.042 — Why this answer works

No. Measure the workload. Check units and whether the load generator itself limited the test.

Visual description:
No. Measure the workload. Check units and whether the load generator itself limited the test.

00:01:13.500 — Check 1

Test bursts and normal load.

Visual description:
Test bursts and normal load.

00:01:18.333 — Check 2

Check units in capacity estimates.

Visual description:
Check units in capacity estimates.

00:01:23.833 — Check 3

Record fallback and test limitations.

Visual description:
Record fallback and test limitations.

00:01:29.333 — Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

Visual description:
Open the lesson
Try the new case
Check and revise
The full task and its requirements are in the linked lesson.