Transformer and inference internals
Explain why longer inputs and outputs can change serving cost and timing.
Reviewed: 2026-10-03
Source-based teaching example · Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — What you will learn

Explain why longer inputs and outputs can change serving cost and timing.

Visual description:
Explain why longer inputs and outputs can change serving cost and timing.

00:00:07.292 — The idea

Processing the input and generating the output are different parts of inference. Saved attention data can reduce repeated computation during generation. Longer context still affects resource use.

Visual description:
Processing the input and generating the output are different parts of inference. Saved attention data can reduce repeated computation during generation. Longer context still affects resource use.

00:00:22.875 — One useful term

Inference. Running a trained model on the inputs selected for a request.

Visual description:
Running a trained model on the inputs selected for a request.

00:00:30.833 — A worked example

The local simulator changes input and output length. At one point, larger context exceeds the response-time target. Its cache estimates explain the tradeoff; they are not hardware benchmarks.

Visual description:
The local simulator changes input and output length. At one point, larger context exceeds the response-time target. Its cache estimates explain the tradeoff; they are not hardware benchmarks.

00:00:47.000 — Your turn to think

Does a longer reasoning run guarantee a true answer? Pause and decide before continuing.

Visual description:
Does a longer reasoning run guarantee a true answer?
Pause the video if you need more time.

00:01:03.583 — Why this answer works

No. More computation is not a truth check. Verify source support, assumptions and the actual supported controls.

Visual description:
No. More computation is not a truth check. Verify source support, assumptions and the actual supported controls.

00:01:15.250 — Check 1

Keep future tokens hidden in the causal example.

Visual description:
Keep future tokens hidden in the causal example.

00:01:21.083 — Check 2

State estimation assumptions and units.

Visual description:
State estimation assumptions and units.

00:01:26.625 — Check 3

Do not present simulation as measured hardware performance.

Visual description:
Do not present simulation as measured hardware performance.

00:01:33.292 — Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

Visual description:
Open the lesson
Try the new case
Check and revise
The full task and its requirements are in the linked lesson.