Attention can connect every token and still lose the sentence
Self-attention removed the recurrent chain, but that creates a new problem: by itself, it is permutation-invariant. If you shuffle the input tokens, the outputs are shuffled in the same way. The mechanism can compare content, yet it has no built-in reason to know which word came first.
A transformer therefore adds position information to token representations. The original architecture uses fixed sinusoidal encodings with sine and cosine waves at different frequencies. These patterns let the model work with relative positions through linear combinations and can extend beyond the sequence lengths seen in training.
BERT instead learns a position embedding table. Learned positions are simple and work well, but they do not naturally generalize beyond the maximum training length. Llama, Mistral, and GPT-NeoX use rotary position embeddings, or RoPE, which rotate query and key vectors by position and support long-context extrapolation better.
Choice: fixed sinusoidal positions can extrapolate, learned positions are simple but length-bound, and RoPE is the dominant modern choice when long-context behavior matters.
An encoder turns every token into a contextual representation
A transformer encoder repeats the same layer structure. Each layer first applies multi-head self-attention, then a residual connection with layer normalization, then a position-wise feed-forward network, followed by another residual connection and normalization.
The attention sublayer lets each token collect information from all positions. The feed-forward sublayer then transforms each position independently with the same learned network. Repeating these operations makes every output token contextual: its final vector reflects the whole input rather than only the token's lookup embedding.
Depth varies by model. The original transformer encoder used 6 layers, BERT-base uses 12, and BERT-large uses 24. The value of the stack is not simply more attention; each layer alternates communication across positions with local feature transformation.
Encoder output preserves one vector per input token. This makes encoder stacks a natural base for understanding tasks such as classification, named-entity recognition, and extractive question answering.
A decoder must learn without seeing the answer ahead
A transformer decoder also uses repeated layers, but generation adds a constraint: when predicting the next token, the model must not look at future target tokens. During training the complete target sequence is available, so a causal mask removes attention to later positions. At inference, the constraint is automatic because those tokens do not exist yet.
In the original encoder-decoder transformer, each decoder layer has masked self-attention, cross-attention over encoder outputs, and a feed-forward sublayer, with residual connections and normalization around each. Decoder queries use cross-attention to retrieve any relevant source position through the encoder's keys and values.
This separation suits transformations such as translation. The encoder can read the entire source bidirectionally, while the decoder generates the target from left to right without cheating.
Causal masking changes what information is available, not the layer's learned weights. The same training batch can contain full target sequences while each position remains restricted to its past.
Deep stacks need a reliable path for gradients
Attention explains how tokens exchange information, but it does not explain how a 12- or 24-layer model trains reliably. Residual connections add a sublayer's input back to its output. This creates a gradient highway through the stack, allowing deep transformers to converge where an unassisted stack would struggle.
Layer normalization stabilizes each token across its feature dimension. Batch normalization instead relies on the batch dimension, which is awkward for variable-length sequences and small batches. LayerNorm works independently for each token, batch size, and sequence length.
The original transformer uses Post-LN, normalizing after the residual addition. Models from GPT-2 onward commonly use Pre-LN, normalizing before the sublayer. Pre-LN is more stable during training and depends less strongly on learning-rate warmup.
Tradeoff: Post-LN matches the original architecture, but Pre-LN has become common because stable optimization is more valuable as transformer depth increases.
Attention routes information; feed-forward layers transform it
Every transformer layer includes a position-wise feed-forward network, usually two linear layers with an activation between them. The hidden width is often four times the model dimension. In BERT-base, a model dimension of 768 expands to a feed-forward dimension of 3072 before projecting back down.
These networks apply the same transformation separately to every token position. They hold a large share of a transformer's parameters and are believed to be where much of its stored knowledge resides. Attention decides which information arrives; the feed-forward sublayer changes the representation that carries it onward.
The original form uses ReLU, while BERT uses GELU. Many modern models use SwiGLU, which gates a Swish-transformed projection with a second projection. The source notes report better performance than ReLU or GELU, with a two-thirds scaling adjustment used to keep parameter counts comparable.
Design balance: increasing attention capacity improves communication among tokens; increasing feed-forward capacity expands per-token transformation and accounts for a large fraction of model parameters.
Architecture determines what context a model can use
The transformer is a system of complementary parts. Positional encodings restore order, attention connects tokens, feed-forward layers transform representations, and residual connections with LayerNorm make deep stacks trainable.
How you arrange the stack changes the task. An encoder can use both left and right context to build representations for every input token. A decoder uses causal masking for left-to-right generation. An encoder-decoder combines full-source understanding with autoregressive output through cross-attention.
Architecture alone does not produce a useful language model. In the next lesson, we will choose a self-supervised objective, train on large unlabeled corpora, and adapt the resulting model. Those choices lead to BERT-style encoders, GPT-style decoders, and T5 or BART encoder-decoders.