The strongest models need more text than people can label
A transformer has the capacity to model language, but a task-specific dataset rarely contains enough examples to teach syntax, semantics, world knowledge, and every useful language pattern from scratch. Human annotation is the bottleneck. The breakthrough was to make the text provide its own supervision.
In self-supervised language modeling, a model predicts content hidden from it or predicts the next token. Learning those objectives forces it to build reusable representations from unlabeled corpora. BERT trained on Wikipedia and BooksCorpus, about 16 GB in the source notes, while later GPT training mixes used hundreds of gigabytes to terabytes from sources including Common Crawl.
Pre-training changes the economics of NLP: expensive labeled examples are reserved for adaptation rather than basic language acquisition. The same foundation can then support classification, extraction, question answering, or generation.
Why before how: model architecture supplies capacity, but the objective determines what information training rewards. Bidirectional understanding and left-to-right generation emerge from different information constraints.
What the model is allowed to see shapes what it becomes
Masked language modeling, used by BERT, hides 15 percent of input tokens and asks the model to recover them from both left and right context. Of selected positions, 80 percent become [MASK], 10 percent become a random token, and 10 percent remain unchanged. The mix prevents the model from learning only how to respond to a special mask symbol.
This objective suits understanding. To recover a hidden word in a sentence about depositing money, the model must build a different contextual representation than it would near a river. Bidirectional evidence is available at every layer.
Causal language modeling, used by GPT, predicts each next token from only the tokens before it. Its main strength is alignment: the training objective is the deployment behavior used for generation. Because a prompted task is also a token sequence, a sufficiently trained causal model can often perform from instructions or in-context examples without changing its weights.
Choice: masked objectives favor rich bidirectional representations for understanding; causal objectives naturally support open-ended generation. Neither objective is universally better—the task decides which information pattern you need.
Pre-training is valuable because it can be reused
The first transfer pattern treated a pre-trained network as a frozen feature extractor. Its hidden states fed a task-specific head, but the base weights never changed. BERT-era full fine-tuning improved results by updating every pre-trained weight on labeled task data.
GPT-3 showed another route: describe the task as text completion and provide examples inside the prompt. In-context learning changes behavior without gradient updates because the context window acts as working memory.
The current default in the source notes is parameter-efficient fine-tuning such as LoRA. Small adapter modules are trained while most base weights stay frozen, reducing compute and limiting catastrophic forgetting. The broader lesson is that adaptation ranges from no weight changes, through small learned modules, to updating the full network.
Tradeoff: feature extraction is cheap but usually weaker; full fine-tuning offers strong adaptation at high storage and compute cost; prompting needs no training; PEFT aims for task adaptation with only a fraction of full fine-tuning compute.
BERT and its successors specialize in understanding
BERT is a stack of transformer encoders trained with masked language modeling and Next Sentence Prediction. BERT-base has 12 layers, 110 million parameters, a model dimension of 768, 12 attention heads, and a feed-forward dimension of 3072. A special [CLS] token is prepended so its final state can serve as a sequence-level representation for classification.
BERT uses a 30,000-item WordPiece vocabulary. Rare words split into subword pieces, such as “un,” “##believ,” and “##able,” reducing the unknown-word problem. Encoder outputs support sequence classification, per-token named-entity labels, extractive question-answering spans, and sentence representations—though Sentence-BERT is preferred over vanilla BERT for sentence similarity.
RoBERTa improves the training recipe with more data, longer training, larger batches, dynamic masking, full sentences, and no Next Sentence Prediction. The notes call RoBERTa-base the de facto baseline encoder. DeBERTa separates content and position in attention and is a quality-first choice, while ALBERT shares parameters across layers and factorizes embeddings to reduce memory. DistilBERT is usually the first practical choice when size reduction is the main goal.
Selection: start with RoBERTa for a general encoder baseline, choose DeBERTa when maximum quality matters more than speed, and consider ALBERT or DistilBERT when memory or latency dominates.
GPT turns one training objective into a general generation interface
GPT-style models use only the decoder stack: masked self-attention followed by feed-forward layers, with no encoder and no cross-attention. GPT-2 ranged from 124 million to 1.5 billion parameters and showed that scaling a decoder-only model produced qualitatively better language generation.
GPT-3 reached 175 billion parameters and made in-context learning visible at scale. A few input-output examples in the prompt can define a task without weight updates. The source notes describe the main hypothesis: diverse pre-training teaches the model to recognize example-then-answer patterns and infer the task structure.
Scaling laws made model improvement more predictable, but Chinchilla revised the allocation. For a fixed compute budget, its result is roughly 20 training tokens per parameter. GPT-3 was undertrained by this standard. The practical implication is to train longer rather than only make a model larger; modern Llama and Mistral models follow the pattern of smaller models trained on more data.
Decoder-only wins at scale through one simple stack, natural in-context learning, and a straightforward KV cache. Its causal context is ideal for generation but does not provide the bidirectional source encoding used by understanding-focused models.
T5 and BART keep input understanding separate from output generation
Encoder-decoder models combine a bidirectional source representation with an autoregressive target. T5 frames every task as text to text: a translation prompt produces translated text, while a sentiment prompt can produce the string “positive.” Its span-corruption objective replaces sampled spans, averaging three tokens, with sentinel tokens and trains the decoder to recover only the missing content.
BART uses denoising pre-training. It corrupts text through masking, deletion, text infilling, sentence permutation, or document rotation, then reconstructs the clean original. That objective resembles summarization and translation: read an altered or source document, then generate a coherent target. BART-large uses 12 encoder and 12 decoder layers with 400 million parameters.
This family remains strong when input and output have distinct roles: machine translation, summarization, structured generation, and fixed-output-length tasks. The encoder sees the full source, and decoder cross-attention can retrieve any source position while generating.
Choice: use Flan-T5 as an instruction-tuned improvement over base T5. Reach for BART when denoising-style generation, especially summarization, matches the task structure.
Choose the family by the flow of information
Model selection becomes clearer when you ask what each output must see. Use an encoder such as RoBERTa or DeBERTa when the task is understanding or extraction and every input token may use both directions. Use a decoder-only model such as GPT, Llama, or Mistral when open-ended generation and prompting are central.
Use an encoder-decoder such as T5 or BART when a complete source must be transformed into a distinct target, as in translation or summarization. Decoder-only models dominate much generation at scale because one stack is simpler to train and serve, in-context examples fit naturally as prefixes, and KV caching is simpler. Encoder-decoders retain an architectural advantage when source and target play clearly different roles.
These four lessons trace the central shift in NLP: from sparse word counts, to dense static meaning, to sequence memory, to direct attention, and finally to transformer families trained on self-supervised objectives. But every one of those families still begins with the same hidden choice: how raw text is split into the tokens the model can actually process. In the next lesson, we will examine that tokenizer boundary and why vocabulary design quietly changes context length, multilingual quality, and cost.