A sequence model must remember what is no longer in view
A static word vector does not change with its sentence, so an early solution was to process text one token at a time and carry a hidden state forward. A recurrent neural network updates that memory as h_t = tanh(W_h × h_{t-1} + W_x × x_t + b). In principle, the current state can summarize everything the model has read.
The difficulty appears when the useful clue is far back. Training unfolds the RNN across time, and gradients are repeatedly multiplied by the recurrent weight matrix. When its largest eigenvalue is below 1, gradients shrink exponentially; above 1, they explode. Gradient clipping can control explosion, but a vanishing signal means the model cannot learn the dependency at all.
Imagine “The cat, which” followed by a fifty-word clause and then “was hungry.” A vanilla RNN may reach “hungry” after losing the information that “cat” was the subject. Sentence-length tasks can work, but dependencies spanning a long document expose the weakness of a single repeatedly updated state.
Tradeoff: recurrent processing naturally handles variable-length and streaming input, but it creates a long computational path between distant tokens. The longer the path, the harder the dependency is to learn.
LSTMs create a path for information to survive
An LSTM adds a cell state alongside the hidden state. You can think of the cell as a conveyor belt whose contents change through mostly additive updates. This gives gradients a clearer path backward than the repeated multiplicative updates of a vanilla RNN.
Three learned gates control that path. The forget gate decides what fraction of the cell to keep, the input gate decides what new information to write, and the output gate decides what part to expose as the hidden state. Initializing the forget gate near 1 lets information flow by default, which directly addresses the fading-memory problem.
A GRU compresses the idea into two gates and removes the separate cell state. It is somewhat faster and performs comparably on many tasks. The source notes recommend LSTMs when longer sequences matter and GRUs when efficiency matters more.
Choice: gating improves memory but does not remove sequential computation. Every state still depends on the previous state, limiting parallelism and leaving long inputs expensive to traverse.
Seeing both directions creates a new bottleneck
A left-to-right model cannot use words that come later. A bidirectional RNN runs one LSTM in each direction and concatenates their hidden states at every position. Each token can then use its full surrounding context. This made bidirectional LSTMs the dominant pre-transformer architecture for sequence labeling.
For tasks that transform one sequence into another, the classic encoder-decoder design uses two RNNs. The encoder reads the source and compresses it into its final hidden state; the decoder then generates the target one token at a time from that context vector.
Compression is the problem. A short sentence may fit into one fixed-size vector, but a long source cannot preserve every detail there. Translation makes the failure concrete: when the decoder needs one source word late in generation, it has only a lossy summary of the entire input.
Still relevant: a BiLSTM with a CRF can remain competitive for named-entity recognition on edge devices. RNNs also fit microcontrollers, streaming inference, explicit sequential state machines, and research where hidden-state interpretation matters.
Let each output retrieve what it needs
Attention keeps every encoder hidden state instead of forcing the source into one vector. At each decoder step, the model scores how relevant each source position is, applies a softmax to those scores, and forms a weighted sum. The decoder receives a context vector chosen for the output it is producing now.
When translating the financial sense of “bank,” for example, the decoder can place high weight on both “bank” and “money.” At another output step, it can focus elsewhere. The source is no longer a fading summary; any encoded token can directly influence any generated token.
Bahdanau attention computes an alignment with a learned feed-forward network: v^T × tanh(W_1 × h_t + W_2 × h_s). Luong attention uses a matrix-weighted or plain dot product. Bahdanau can express asymmetric alignments, while the dot-product form is cheaper and scales better. Scaled dot-product attention became the foundation of the transformer.
Tradeoff: additive attention is slightly more expressive, while multiplicative attention is computationally simpler. Architecture choices increasingly favored the version that could scale.
Every token can speak directly to every other token
Self-attention applies the same retrieval idea within one sequence: queries, keys, and values all come from that sequence. Each token learns which other positions matter and aggregates their information. Distance no longer creates a chain of recurrent updates, so long-range interaction is available by construction.
Multi-head attention repeats this operation in parallel with different learned projections, concatenates the head outputs, and projects them again. Different heads can specialize in different relationship types, including syntactic dependencies, coreference, and positional structure. The original transformer used 8 heads with a model dimension of 512; BERT-base uses 12 and BERT-large uses 16.
This flexibility has a price. Full self-attention compares every token pair, so its complexity is O(n²) in sequence length. At 512 tokens, that is 262,144 pairwise attention computations per layer per head. Efficient attention methods matter because longer context makes this cost rise quickly.
Tradeoff: self-attention shortens the path between distant tokens and enables parallel sequence processing, but quadratic pairwise comparisons make long context expensive.
Attention solved access; the transformer builds the rest
Sequence models introduced memory, and gates helped that memory survive. Attention made a more decisive change: information no longer had to travel through every intervening state or fit into one source summary. A token could retrieve another token's representation directly.
But attention alone does not tell a model which token came first, how to train a deep stack reliably, or how to transform each attended representation. Those needs are handled by positional information, residual connections, normalization, and feed-forward layers.
The next lesson assembles those pieces into the transformer architecture and shows how encoder and decoder stacks support different kinds of NLP work.