Watch the explanation. Practice the skill.
Transformer and inference internals
Explain why longer inputs and outputs can change serving cost and timing.
Captions appear here during playback.
Transformer and inference internals
Explain why longer inputs and outputs can change serving cost and timing.
Video playback is unavailable. Read the descriptive transcript below, go directly to practice, or open the video file.
Source-based teaching example · Fictional data · Reviewed 2026-10-03. Microsoft Mark · locally synthesized. On a small screen, use fullscreen for diagram details or read the transcript.
Read the descriptive transcript
0:00 · What you will learn
Explain why longer inputs and outputs can change serving cost and timing.
What appears on screen
Explain why longer inputs and outputs can change serving cost and timing.
0:07 · The idea
Processing the input and generating the output are different parts of inference. Saved attention data can reduce repeated computation during generation. Longer context still affects resource use.
What appears on screen
Processing the input and generating the output are different parts of inference. Saved attention data can reduce repeated computation during generation. Longer context still affects resource use.
0:22 · One useful term
Inference. Running a trained model on the inputs selected for a request.
What appears on screen
Running a trained model on the inputs selected for a request.
0:30 · A worked example
The local simulator changes input and output length. At one point, larger context exceeds the response-time target. Its cache estimates explain the tradeoff; they are not hardware benchmarks.
What appears on screen
The local simulator changes input and output length. At one point, larger context exceeds the response-time target. Its cache estimates explain the tradeoff; they are not hardware benchmarks.
0:47 · Your turn to think
Does a longer reasoning run guarantee a true answer? Pause and decide before continuing.
What appears on screen
Does a longer reasoning run guarantee a true answer? Pause the video if you need more time.
1:03 · Why this answer works
No. More computation is not a truth check. Verify source support, assumptions and the actual supported controls.
What appears on screen
No. More computation is not a truth check. Verify source support, assumptions and the actual supported controls.
1:15 · Check 1
Keep future tokens hidden in the causal example.
What appears on screen
Keep future tokens hidden in the causal example.
1:21 · Check 2
State estimation assumptions and units.
What appears on screen
State estimation assumptions and units.
1:26 · Check 3
Do not present simulation as measured hardware performance.
What appears on screen
Do not present simulation as measured hardware performance.
1:33 · Now try a different case
Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.
What appears on screen
Open the lesson Try the new case Check and revise The full task and its requirements are in the linked lesson.
Watching explains the example. Complete the lesson practice to check what you can do. Explore all visual lessons →