Your model never sees words
You can choose a strong architecture and still lose useful context before the first layer runs. The model does not receive words or sentences. It receives integer token IDs produced by a tokenizer, and that conversion decides how much text fits in the context window and which patterns are easy to learn.
Subword tokenization sits between brittle word vocabularies and raw characters. Common words can remain whole tokens, while rare words break into reusable pieces. That largely removes the out-of-vocabulary problem without forcing the model to process every character separately.
This is why tokenization is an architecture decision rather than a cleanup step. Its vocabulary, training corpus, and segmentation rules shape memory use, multilingual coverage, and the effective length of every input.
Tradeoff: whole-word vocabularies are compact for familiar text but fail on unseen words. Character vocabularies cover almost anything but create long sequences. Subwords trade a larger vocabulary for shorter, reusable sequences.
Build useful pieces by merging what repeats
Byte Pair Encoding starts with characters and an end-of-word marker. It counts adjacent pairs across the corpus, merges the most frequent pair into a new symbol, then repeats until the vocabulary reaches its target size.
Imagine training on “low,” “lower,” “newest,” and “widest.” Frequent pairs can merge from “e” plus “s” into “es,” then “es” plus “t” into “est.” With enough merges, common words become single tokens while uncommon words remain combinations of learned pieces.
GPT-2 uses byte-level BPE. Because it begins with raw bytes rather than a fixed set of Unicode characters, it can encode any input without an unknown token. Coverage is guaranteed, though unusual text may still expand into many tokens.
Tradeoff: BPE chooses frequent pairs, not necessarily linguistically meaningful units. It guarantees a segmentation, but the pieces may not line up with prefixes, roots, or suffixes.
Similar goal, different search strategy
WordPiece, used by BERT, resembles BPE but selects merges that improve language-model likelihood rather than simply choosing the most frequent pair. Continuation pieces carry the “##” marker, so “unbelievable” might become “un,” “##believ,” and “##able.”
The Unigram Language Model takes the opposite route. It starts with a large candidate vocabulary and repeatedly prunes weak pieces. A word can have several valid segmentations, and a learned unigram model chooses the most likely one.
SentencePiece implements the unigram approach and does not require spaces to define word boundaries. That makes it useful for languages without consistent whitespace and for multilingual training where one set of preprocessing assumptions does not fit every language.
Tradeoff: WordPiece gives a stable best segmentation but is slower to train than frequency-based BPE. Unigram tokenization supports several plausible segmentations, which adds flexibility but also a probabilistic training step.
Vocabulary size moves cost between the embedding table and the sequence
A small vocabulary such as BERT’s 30,000 entries keeps the embedding table smaller, but it splits more words into multiple tokens. A vocabulary of 100,000 to 250,000 entries consumes more memory, yet often represents sentences with fewer tokens and gives broader multilingual coverage.
The useful measure is fertility: the average number of tokens produced per word. An English-trained tokenizer usually has low fertility on English. The same tokenizer can have high fertility on Turkish, Finnish, Chinese, or Thai because morphology, scripts, and spacing differ from its training distribution.
Context windows are measured in tokens, not words. A 512-token encoder might fit roughly 300 English words but only about 200 Turkish words. The source notes give an even sharper example: LLaMA’s 32,000-entry BPE vocabulary has about twice the fertility on Chinese as on English, so the same nominal window processes about half as much Chinese content.
Tradeoff: larger vocabularies lower fertility and improve multilingual coverage, but they require larger embedding and output matrices. Smaller vocabularies save parameter memory while spending more attention compute on longer sequences.
Do not teach the model that one split is the only split
A deterministic tokenizer always gives a word the same segmentation. That is convenient, but it lets the model become dependent on one arbitrary split. Subword regularization changes the training signal by sampling among valid tokenizations instead of always choosing the most likely one.
The word reaches the model through several plausible piece sequences during training. The model must learn a representation that survives those variations rather than memorizing one boundary pattern. SentencePiece supports this behavior directly.
This matters most when spaces are optional, compound words are common, or spelling varies. It is a training technique, not an inference requirement: deployment can still use the most likely segmentation for stable behavior.
Measure the text your system will actually process
The tokenizer decides what the model can represent, how quickly a context window fills, and which languages pay an extra token cost. BPE builds pieces through frequent merges, WordPiece optimizes a language-model objective, and unigram tokenization prunes a large vocabulary while preserving alternative segmentations.
Do not choose among them by name alone. Tokenize samples from your real languages and domains, measure fertility, inspect how specialist terms split, and account for the memory cost of the vocabulary. Those observations turn tokenization from an inherited default into an explicit engineering choice.
Once text has become token IDs, the next constraint is learning from them without unstable gradients or impossible memory demands. In the next lesson, we will connect training schedules, precision, distributed execution, and parameter-efficient adaptation into one practical training workflow.