LLM internals: how transformers work

Lesson preparation and outcomes

Prerequisites: Vector concepts and inference versus training.

By the end, you can: Explain causal attention, KV reuse and context/state distinctions.

Worked scenario and practice feedback

Four causal tokens permit 1+2+3+4=10 query/key pairs. A simplified KV estimate is 2 x layers x KVheads x head_dimension x bytes x tokens; 2/2/4/2/10 gives 640 bytes. Grouped-query architectures and serving overhead change actual memory.

Guided and independent practice: Draw the triangular mask and compute the estimate. Independently double token count, then list decisions and pending actions that must survive lossy transcript compaction externally.

Answer criteria: No future key visible; units correct; no assumption every model has identical architecture or remembers every token; supported decoding controls checked. If the local pack or interpreter is unavailable, trace the synthetic input, expected output and failure guard on paper; label execution unverified. Required practice uses no paid API calls, accounts, purchase, operational data or actual deployment.

Transformers and transformer-derived designs are important language-model architectures. Hybrids and other designs also exist; provider internals may be undisclosed. Learn this architecture without assuming every system uses the identical mechanism. You don’t have to be able to build one to be effective with it, but you should be able to explain how it works on a whiteboard. This lesson is that whiteboard explanation.

Part 1 The job the transformer is doing.

A language model is given a sequence of tokens (roughly, words) and asked to predict the next token. That’s the whole job at the architectural level. Everything else (chat, code generation, agents, RAG) is built on top of “predict the next token” repeatedly.

Transformers provide a widely studied way to condition predictions on token representations; this lesson explains the mechanism rather than claiming a universal architectural winner.

Three reasons it works:

• Parallelism. Earlier architectures (RNNs) had to process tokens one at a time. Transformers process the whole sequence at once during training. This is why scaling worked: you can train on much more data in much less wall-clock time.
• Long-range context. Dense causal attention can connect positions within the available context; sliding or sparse variants restrict that pattern. Access to earlier tokens does not guarantee reliable recall, reasoning or correct use.
• Scaling that holds up. Scaling studies describe relationships under specific data, compute and optimization assumptions. More parameters or context do not guarantee improvement on every task; evaluate the actual system.

Part 2 Tokens and embeddings: the input side.

The model doesn’t see words. It sees tokens: integer IDs.

Tokenization

A tokenizer splits text into tokens. The default in modern LLMs is byte-pair encoding (BPE) or its variants. BPE merges frequent character pairs into single tokens; common words become single tokens, rare words become multiple tokens.

# Conceptually:
tokenize("VetsLearn") -> ["Vets", "Learn"]  # invented illustrative split, not tokenizer output
tokenize("Antidisestablishmentarianism") -> ["Anti", "dis", "establish", "ment", "arian", "ism"]   # 6 tokens

Token/word ratios vary by tokenizer, language, whitespace and content. The illustrative splits above are not measurements; use an actual selected tokenizer to count.

Embeddings

Each token ID is mapped to an embedding: a vector of floating-point numbers (often 1024 to 12288 dimensions in modern LLMs). The embedding is the model’s representation of what that token means.

Embeddings are learned during pre-training. Tokens with similar meanings end up with similar embeddings. The classic example: the vector for “king” minus the vector for “man” plus the vector for “woman” lands near the vector for “queen.” Modern embeddings are richer than this party trick suggests, but the intuition is right.

Positional encoding

Embeddings tell you what a token means, not where it is. Some designs add absolute position vectors to token representations. Rotary position embeddings instead rotate query/key coordinates by position, yielding relative-position effects in attention; they are not merely vectors added to input embeddings. The exact position mechanism depends on architecture.

Part 3 Attention: the core mechanism.

Attention is the move that makes transformers work. Internalize it.

For each token, attention asks: “Given the previous tokens in the sequence, which ones should I pay attention to right now, and how much?”

The Q, K, V triplet

For each token, the model computes three vectors from its embedding:

• Query (Q). “What am I looking for?”
• Key (K). “What can I be matched on?”
• Value (V). “What information do I carry if you decide to attend to me?”

To compute attention for token i:

1. Take i’s query vector.
2. Compute query/key dot products and divide by sqrt(d_k) for scaled dot-product attention, where d_k is key dimension. Apply the causal mask before normalization.
3. Softmax the scores so they sum to 1 (these are attention weights).
4. Sum every token’s value vector weighted by its attention weight.

The result is a new representation for token i that has pulled in information from across the sequence, weighted by what was relevant.

That paragraph is the whole secret. Re-read it.

Multi-head attention

Real transformers do this many times in parallel, with different Q/K/V projections. Each “head” can specialize: one might track syntactic structure, another semantic similarity, another long-range dependency. The outputs are concatenated and projected back to the model’s hidden dimension.

Causal masking

For language modeling, you can’t let token i attend to tokens that come after it (that would be cheating; the model would see the answer). The attention scores for future tokens are set to negative infinity before the softmax, which makes their weights zero. This is the “causal mask.”

Part 4 The full layer.

A transformer layer is attention + a feedforward network, both wrapped in normalization and residual connections.

In pseudocode:

def transformer_layer(x):
    # x: input embeddings for the sequence
    # Self-attention block
    attn_out = self_attention(layer_norm(x))
    x = x + attn_out                         # residual

    # Feedforward block
    ff_out = feedforward(layer_norm(x))
    x = x + ff_out                           # residual
    return x

The feedforward is typically a two-layer MLP that’s about 4x wider than the model’s hidden dimension. It transforms each token independently. Attention mixes information across tokens; feedforward processes each token’s representation.

A modern LLM stacks 32 to 100 of these layers. Each layer refines the representations a little. By the top of the stack, the representations for each token are heavily contextualized: they capture what the token means in the specific surrounding text, not just in isolation.

The output of the top layer goes through one final linear projection back to the vocabulary size, producing a probability distribution over what the next token might be.

Part 5 Decoding: how the model produces output.

At inference time, the model produces output one token at a time:

1. Take the prompt, tokenize, embed.
2. Run through the transformer.
3. The output at the last position is a probability distribution over the vocabulary.
4. Sample (or pick) a token from that distribution.
5. Append the sampled token to the sequence.
6. Repeat from step 2 with the new sequence, until you hit a stop token or max length.

That’s “autoregressive generation.” Each output token depends on all the previous tokens.

Sampling strategies

• Greedy. Always pick the highest-probability token. Chooses the highest-scoring token for fixed logits; serving nondeterminism and ties can still affect reproducibility.
• Temperature. A knob that flattens (high temperature) or sharpens (low temperature) the probability distribution before sampling. Higher values flatten the distribution and increase sampling variation; lower values sharpen it. This changes sampling, not factual truth.
• Top-k. Sample only from the top k most probable tokens.
• Top-p (nucleus sampling). Sample only from the smallest set of tokens whose cumulative probability exceeds p.

Supported decoding controls depend on model and endpoint; some reasoning models restrict them. Check the capability manifest rather than passing every familiar parameter. JSON mode checks syntax; supported schema modes constrain structure; semantic validation and evidence checks remain necessary.

KV cache

A practical detail that matters for production: at each step of generation, the keys and values for prior tokens don’t change. You cache them. KV caching reuses prior key/value projections during decoding, while attention to cached positions still consumes work and memory. It is distinct from prompt caching and persistent conversation memory. Streaming can improve time to first content; it does not guarantee shorter total runtime.

Interface/reference limits

Use the workflow reference to distinguish model capability, deployment and harness controls. Provider snippets in this lesson are illustrative, have not been executed against an actual provider in this revision, and require current SDK/model capability verification before separately authorized use.

Primary source checked October 2, 2026: OpenAI reasoning controls and state guidance.

Primary architecture sources: Attention Is All You Need and RoFormer, checked October 2, 2026. They explain these mechanisms and do not disclose every commercial model architecture.