A model can win the metric and still fail the product
You can train the right architecture for the right task and still make the wrong deployment decision. An automatic metric may reward surface overlap instead of meaning. A compressed model may retain benchmark quality but miss the rare class that matters. A fast decoder may still run out of memory because its cache grows with every token and every request.
Evaluation and efficiency therefore belong in the same design loop. First define what a correct output means. Then measure it with a metric that exposes the costly failures. Only after that should you compress or accelerate the model, using the same measurement to verify what you preserved.
There is no single score or optimization that settles the choice. The goal is a system whose quality, latency, memory, and cost fit the task at the same time.
Tradeoff: optimizing one headline metric makes decisions easy, but hides failures outside that metric. A useful evaluation set keeps several views of quality and ties them to the deployment constraints you actually face.
Measure the kind of agreement the task requires
BLEU measures n-gram precision against references and applies a brevity penalty. ROUGE is recall-oriented, with ROUGE-L using the longest common subsequence, so it became standard for summarization. Both are useful when reference wording matters, but both can reject a valid paraphrase: “The car is fast” can score badly against “The automobile is quick.”
METEOR adds stemming and WordNet synonym matching. BERTScore goes further by comparing contextual token embeddings and computing an F1 score over cosine-similarity matches. BLEURT fine-tunes BERT to predict human quality ratings, improving correlation with human judgments at the cost of loading a model of roughly one to four gigabytes.
For classification, precision asks how many predicted positives were correct; recall asks how many actual positives were found; F1 balances the two. Macro F1 weights each class equally, while micro F1 lets large classes dominate. For NER, use entity-level F1 so the full span must match, not the easier token-level score.
Tradeoff: lexical metrics are cheap and reproducible but miss semantic equivalence. Learned metrics capture meaning better but add model cost and can inherit model bias. Keep the metric aligned with the output contract rather than using the most modern name by default.
Know when the number stops answering the question
Perplexity measures a language model’s average surprise at the next token. Lower is better, and cross-entropy loss is the logarithm of perplexity. A perplexity of 50 can be read as the model choosing among about 50 equally likely next tokens on average.
But perplexity is only comparable when tokenization and vocabulary are the same. You cannot directly compare GPT-2 and T5 perplexity, and low surprise does not guarantee factual correctness. The metric evaluates token prediction, not whether a statement is true or useful.
Open-ended generation eventually needs judgment. BLEU and ROUGE correlate poorly with human preference when many paraphrases, styles, or dialogue responses can be valid. Human evaluation is appropriate for subjective architecture comparisons and important final decisions. An LLM judge is cheaper and often correlates better than n-gram metrics, but it may prefer its own style or longer answers, so the notes recommend averaging judgments from multiple models.
Tradeoff: human evaluation is closest to subjective product quality but is slow and expensive. LLM judges scale more cheaply, yet introduce preferences of their own. Neither should be disguised as an objective ground truth.
Reduce what you store before optimizing how it runs
Knowledge distillation trains a smaller student to reproduce a larger teacher’s probability distribution, not only its hard labels. Those soft probabilities carry relationships between alternatives. DistilBERT applies this idea to reach 66 million parameters, run 40 percent faster than BERT-base, and retain 97 percent of its GLUE performance.
Quantization reduces numerical precision. INT8 post-training quantization cuts FP32 weight memory by four with near-zero quality loss on many tasks. GPTQ uses calibration data to minimize layer reconstruction error in 4-bit models. AWQ preserves important, highly activated weights at higher precision and outperforms GPTQ on many benchmarks at the same compression ratio.
GGUF and llama.cpp bring mixed-bit quantized models to CPUs and Apple Silicon. The notes give Llama-3-8B on a MacBook Pro as the practical example, at roughly 10 tokens per second compared with more than 80 on a GPU. Pruning can remove individual weights or whole heads and neurons, but unstructured sparsity does not become faster unless the hardware has sparse kernels.
Tradeoff: distillation needs a teacher and a new training run. Quantization is easier after training but may lose quality at lower bit widths. Structured pruning maps to ordinary hardware better; unstructured pruning needs sparse-kernel support to deliver speed.
Move fewer bytes, not just fewer operations
Standard attention materializes the full sequence-by-sequence attention matrix in high-bandwidth GPU memory. FlashAttention tiles the computation into fast SRAM and avoids materializing that matrix. It performs the same mathematics, but changes the memory-access pattern.
That distinction matters because memory bandwidth, not arithmetic, is often the bottleneck. The source notes report two-to-four-times faster attention and ten-to-twenty-times less attention memory, enabling longer-sequence training on the same hardware.
FlashAttention does not remove attention’s quadratic pairwise work. It makes the exact computation much more hardware efficient. FlashAttention-2 and -3 extend these gains to multi-query attention and H100 hardware, and PyTorch exposes the optimized path through scaled dot-product attention.
Tradeoff: FlashAttention improves exact attention without changing model quality, but it does not make an unlimited context free. Sequence length still increases pairwise computation, and the rest of the model still consumes memory.
Generation speed is a cache-management problem
Autoregressive decoding reuses the past at every step. A KV cache stores the keys and values already computed for previous tokens, so the model computes attention only for the new position. Without that cache, it would repeat work across the full prefix for every generated token.
The cache grows linearly with sequence length and batch size. For Llama-3-70B at a 4,000-token context and batch size 32, the notes estimate about 28 gigabytes for KV cache alone. Multi-Query Attention shares one key-value head across query heads; Grouped-Query Attention shares within groups and gives a better quality-to-memory balance. PagedAttention in vLLM allocates cache in virtual-memory-like pages for variable-length, high-throughput serving.
Speculative decoding attacks latency from another direction. A small draft model proposes several tokens, and the target model verifies them in one parallel pass. With a compatible tokenizer and an adjacent-quality draft model, acceptance rates are typically 70 to 90 percent and speedups are two to four times. Rejected tokens still fall back to target-model sampling, so the target distribution is preserved.
Tradeoff: KV caching removes repeated computation but converts long contexts and large batches into a memory problem. Speculative decoding reduces latency, but requires a compatible draft model whose proposals are accepted often enough to repay its own compute.
Choose the simplest system that preserves the signal you need
The course began with sparse counts because sophisticated models are not automatically better decisions. TF-IDF remains useful when vocabulary is the signal, data is small, and interpretation matters. Static embeddings add semantic neighborhoods. Recurrent networks add sequence state. Attention removes the fixed-vector bottleneck, and transformers make direct token-to-token interaction the organizing principle.
From there, architecture follows output. Encoders suit understanding and extraction. Decoder-only models suit open-ended generation and in-context learning. Encoder-decoders remain natural for translation, summarization, and structured transformation. Tokenization decides how much text reaches any of them, while full fine-tuning, LoRA, and QLoRA decide how much of the model moves during adaptation.
The final discipline is to close the loop. Match metrics to the failure you care about, include human judgment where references cannot define quality, and re-run those evaluations after distillation, quantization, pruning, attention optimization, or serving changes. Efficiency is successful only when the required behavior survives.
Your architecture choice is therefore not a contest among model names. It is a chain of explicit constraints: data volume, output shape, language coverage, context length, quality metric, memory, latency, and cost. When each choice answers one of those constraints, you can defend the system—and know what to change when the constraint moves.