The gap between knowing and shipping
You've spent nine lessons learning retrieval patterns — dense embeddings, learned sparse methods, late interaction, RRF fusion, chunking strategies, graph traversal. Each one is a powerful tool. But there's a moment every engineering team hits when they ask: given all of this, what do we actually build?
Production systems are wired together from these components. Some combinations are theoretical curiosities. Others have proven themselves at scale across hundreds of teams. This lesson is about that second category — the pipeline architectures that show up again and again in real deployments, and the reasoning behind why each one exists.
There are five patterns worth knowing, and they occupy different positions on the tradeoff surface between latency, quality, and complexity.
Recall first, precision second
The two-stage pipeline is the near-universal production default — and for good reason. You've seen the same architecture in web search for decades, and it works because it respects a fundamental constraint: fast retrieval and accurate ranking are different problems that require different tools.
Stage one optimizes for recall. You run fast ANN retrieval over your dense index — or a hybrid dense plus sparse pass fused with RRF — and you pull back the top hundred candidates. Speed is the priority. You don't need to be precise here; you need to not miss the right answer.
Stage two optimizes for precision. You take those hundred candidates and run them through a cross-encoder reranker, which scores each document against the query in a much more computationally intensive way. The reranker reads both the query and the document together, attending to their interaction, and returns a ranked list. You return the top ten.
The architecture has a second virtue: each stage can be improved independently. You can swap in a better embedding model without touching the reranker. You can upgrade the reranker without changing your index. Debugging is simpler too — you can inspect what the first stage returned before reranking, which makes failure analysis tractable.
The constraint: The reranker is a cross-encoder, which means it runs inference on every candidate pair at query time. Keeping stage one's output at roughly a hundred documents is important — if you pass a thousand, latency becomes a problem even with GPU acceleration.
The highest-quality path
If the two-stage pipeline is the pragmatic default, RAG-Fusion plus reranking is the pattern you reach for when quality is the primary constraint and two to four seconds of latency is acceptable.
The architecture chains four steps. First, an LLM generates multiple rephrased versions of the incoming query — typically three to five. Then you run retrieval in parallel for all of them simultaneously, so the latency cost of multiple queries is amortized. Next, you fuse the result sets using Reciprocal Rank Fusion, which surfaces documents that appear consistently across multiple query variants. Finally, a cross-encoder reranker scores the fused pool against the original query.
What you gain is diversity. A single query hits one region of the embedding space. Multiple phrasings hit different regions, and their union has substantially higher recall for complex or ambiguous questions. The reranker then applies precision on top of that high-recall pool.
This pattern is especially effective for knowledge-intensive tasks — technical support, legal research, medical question answering — where missing the right document is a meaningful failure mode.
The cost: Each additional query variant is another set of retrieval calls. At five variants plus reranking, total latency typically lands in the two to four second range. That's fine for async or professional workflows, but too slow for real-time conversational interfaces where users expect sub-second responses.
Dense and sparse, side by side
Ensemble retrieval addresses a different gap. Dense retrieval is excellent at semantic matching — it handles synonyms, paraphrases, and conceptual similarity. Sparse retrieval is excellent at exact matching — it doesn't miss a document just because it uses a different word for the same idea in a technical context. Neither does the other's job well.
The ensemble pattern runs both retrievers in parallel, fuses their outputs with RRF, and feeds the fused result set to a cross-encoder reranker. You get exact-match strength and semantic-match strength simultaneously, and RRF handles the combination without requiring you to normalize scores across two fundamentally different scoring systems.
Teams building systems over mixed corpora — documentation mixed with code, structured FAQs mixed with unstructured prose — find this pattern particularly reliable. The failure modes of the two retrievers don't correlate, so the union almost always has higher recall than either alone.
The infrastructure ask: You're running two retrieval systems instead of one. That means maintaining both a vector index and either BM25 or SPLADE. For many teams the quality improvement justifies the operational cost, but it's a real cost to account for in your infrastructure planning.
Not every query deserves the same strategy
The patterns so far assume a single retrieval strategy handles all queries. Routing starts from a different premise: different query types are best served by fundamentally different backends, and the system should classify the incoming query and dispatch accordingly.
A query asking for the current price of a stock is a structured data lookup — the right answer comes from a database query, not a vector search. A broad thematic question like "what are the key themes in Q3 earnings calls" maps well to a global GraphRAG search across a document graph. A specific lookup — "what does section 4.2 of the vendor contract say about liability" — calls for vector search plus a metadata filter on the right document. A question about something that happened last week might require a web search fallback.
The classification step can be rule-based — regex patterns or keyword matching for well-defined query types — or LLM-classified for fuzzier categories. LlamaIndex's RouterQueryEngine implements this pattern and handles the dispatch logic for you. The key design decision is defining the boundary conditions: what happens when a query falls between categories, and how do you handle the fallback?
The complexity tradeoff: Each route is a separate subsystem to build, test, and maintain. Routing makes sense when your system genuinely spans multiple data modalities or backends. If you're working over a single homogeneous corpus, it adds complexity without adding value.
When the corpus might not have the answer
Corrective RAG — CRAG — solves a problem that most retrieval systems quietly ignore: what happens when the retrieval results are bad? In a standard RAG pipeline, poor retrieval silently produces poor generation. The model answers confidently from irrelevant documents. CRAG adds a relevance scoring step after retrieval and before generation, and acts on the score.
If all retrieved documents score above a relevance threshold, the pipeline proceeds normally. If all score below threshold — a sign that your corpus simply doesn't contain a good answer — the system falls back to a web search to find external evidence. If results are mixed, CRAG filters out the low-quality documents and optionally supplements with web search. The result is a system that degrades gracefully instead of hallucinating confidently.
Speculative RAG takes a different angle. It samples a small random subset of retrieved documents, generates a draft answer from that subset, then uses that draft to guide a second targeted retrieval pass — running it in parallel with standard retrieval. The two threads merge before generation. The pattern is still more research than production, but it points at something real: your first retrieval pass is often noisier than it needs to be, and a partial answer can help you ask better follow-up questions.
The honest assessment: CRAG is production-ready and particularly valuable for open-domain systems where your corpus has known gaps — internal knowledge bases that don't cover breaking news, for instance. Speculative RAG is worth watching but not yet the default choice for most teams.
Start with two stages, then layer in complexity
Most production RAG systems start with the two-stage pipeline, and many never need to go further. Recall plus reranking handles a wide range of problems well, and the architecture is debuggable, maintainable, and easy to improve incrementally.
When you need to go further: RAG-Fusion plus reranking is the path to higher quality at the cost of latency. Ensemble retrieval is the path to robustness over mixed corpora. Routing is the path to a system that handles genuinely heterogeneous query types. CRAG is the path to graceful degradation when your corpus has gaps. Each layer adds real value — but also real complexity. Add them deliberately, when you have evidence the simpler approach isn't meeting your quality bar.
In the next lesson, we'll look at how to measure whether any of these choices are actually working — the evaluation frameworks and metrics that let you make evidence-based decisions about your pipeline.