How you split the documents changes everything
Most teams building RAG systems spend the bulk of their time choosing embedding models and tuning retrieval. The chunking pipeline gets set once, usually to some default — 512 tokens with a 10% overlap — and then quietly forgotten. That's a mistake. In many systems, chunking decisions matter more than embedding model choice.
The problem is fundamental: language models have limited context windows, so you can't feed them whole documents. But how you cut a document apart determines what information is recoverable, what gets buried at chunk boundaries, and whether the embedding of any given chunk is coherent enough to be retrieved at all.
There are seven strategies worth knowing. They form a spectrum from simple and cheap to complex and expensive — and the right choice depends on your document types, query patterns, and how much you're willing to invest at indexing time.
The default that mostly works, until it doesn't
Fixed-size chunking is exactly what it sounds like: split the document every N tokens, with an M-token overlap between adjacent chunks so ideas that straddle a boundary aren't completely severed. The standard default — 512 tokens, 50–100 token overlap — isn't based on any deep principle; it just falls within the context windows of early embedding models and produces chunks that embeddings can handle reasonably well.
For well-formatted prose — documentation, blog posts, textbooks — it works adequately. The chunks tend to be coherent enough that embeddings capture the main topic, and the overlap prevents the worst sentence-splitting artifacts. It's also the easiest to implement: no models, no dependencies, just a loop over tokens.
The failure mode emerges with heterogeneous content. A technical document mixing prose paragraphs, code blocks, tables, and headers doesn't chunk gracefully at fixed intervals. A 512-token boundary might land inside a code example, or split a table in half, producing chunks that embed incoherently. The overlap helps at the margin but can't fix a structural mismatch.
The hidden cost: Overlap creates near-duplicate content in your index. Two adjacent chunks share 50–100 tokens of identical text — which means similar queries will retrieve both, and you'll pay twice at the LLM context level. At scale, this adds up to meaningful storage and latency overhead with no retrieval benefit.
Let the content tell you where to cut
Instead of cutting at fixed intervals, semantic chunking asks the document itself where the topic boundaries are. The approach: embed consecutive sentences, then look for places where the cosine similarity between adjacent sentences drops sharply. A sharp drop in similarity is a signal that the text has shifted topics — which is exactly where you want to split.
The result is chunks of variable length that tend to be topically coherent. A three-paragraph digression about authentication stays in one chunk; a transition to a new concept becomes its own chunk. Both LlamaIndex and LangChain implement this out of the box, so the barrier to use is low.
The improvement over fixed-size chunking is most pronounced in mixed-topic documents — anything where different sections cover genuinely different concepts. For single-topic documents like a focused FAQ or a technical spec for one API endpoint, the difference is smaller.
The noise problem: Boundary detection is imperfect. Sentences that happen to use different vocabulary but are part of the same idea can trigger false splits. Sentences with anaphoric references — "it", "this approach", "the above" — embed poorly in isolation and can confuse the similarity calculation. Semantic chunking is better than fixed-size on average, but it's not reliable on short or poorly-written text.
Building a tree so retrieval can work at any altitude
Regular RAG has a resolution problem. When a user asks a broad question — "what's this document's overall argument?" — vector search returns specific chunks, which contain details, not themes. When they ask a specific question, they might need a chunk that only makes sense in context of the surrounding material. You're stuck: too granular for broad queries, too narrow for cross-document synthesis.
RAPTOR — Recursive Abstractive Processing for Tree-Organized Retrieval — solves this by building a hierarchy. Start with your leaf chunks. Cluster them by embedding similarity. Summarize each cluster with an LLM. Embed those summaries and cluster them again. Repeat until you have a single root summary. The result is a tree: leaf nodes are specific chunks, internal nodes are progressively higher-level summaries, the root is the document's big picture.
At retrieval time, you search across all levels. A broad query lands on a high-level summary; a specific one lands on a leaf. Both can be returned together, giving the LLM both the thematic context and the detail it needs to answer. For long documents and cross-document questions where the answer requires synthesizing multiple sections, RAPTOR consistently outperforms flat retrieval.
The indexing cost: Building the tree requires many LLM summarization calls — one per cluster at each level. For a large corpus, this is slow and expensive. RAPTOR is best suited to static or slowly-changing document collections where you can afford to build the index once and query it many times. It's not a good fit for real-time ingestion pipelines.
The most precise retrieval unit: a single fact
What if the unit of retrieval wasn't a paragraph or even a sentence, but a single atomic fact? That's the premise of proposition-based chunking. You run each passage through an LLM with a prompt that asks it to decompose the text into minimal, self-contained factual statements — propositions. Each proposition becomes its own indexed chunk.
The improvement in retrieval precision is substantial. Consider a passage: "Einstein won the Nobel Prize in 1921 for his work on the photoelectric effect." Fixed-size chunking would bundle this with surrounding sentences about other topics. Proposition-based chunking extracts two atomic statements: "Einstein won the Nobel Prize in 1921" and "Einstein's Nobel Prize was awarded for the photoelectric effect." Each can be retrieved independently and precisely for different queries.
Because each proposition is short and contains exactly one idea, its embedding is maximally expressive for that idea — there's no noise from surrounding, unrelated content. For fact-retrieval tasks — "when did X happen?", "what caused Y?", "who is responsible for Z?" — this approach consistently produces the highest precision of any chunking method.
The indexing pipeline cost: You're making one LLM call per passage to generate propositions, then often generating another embedding per proposition. For large corpora, this is the most expensive indexing strategy by a wide margin. The quality gains are real but only justify the cost when retrieval precision is the dominant concern — legal, medical, and compliance use cases tend to be the natural fit.
Retrieve small, return big
There's a tension at the heart of chunk sizing. Small chunks embed more precisely — a 128-token chunk is focused enough that its embedding strongly represents its specific content. But when a small chunk is retrieved and handed to the LLM for generation, it often lacks the surrounding context needed to actually answer the question. The sentence was retrieved correctly, but the answer requires the paragraph.
Parent-child chunking — sometimes called small-to-big retrieval — resolves this tension by separating the retrieval unit from the context unit. You index small child chunks (128 tokens is common) for precise retrieval. But each child chunk is linked to a larger parent chunk (512–1024 tokens) that contains it. When a child is retrieved, you fetch and return the parent to the LLM instead.
The result: retrieval happens at high precision using the small chunk's embedding, but the LLM receives enough surrounding context to construct a coherent answer. LlamaIndex has native support for this pattern with its NodeWithScore and parent-document retriever abstractions, making implementation straightforward.
Storage note: You're storing two versions of every piece of content — the child index and the parent documents. You also need to maintain the child-to-parent mapping. Neither is a heavy burden, but it's worth accounting for in your storage estimates, especially at the hundred-million-chunk scale.
Two ways to make chunks remember their context
Every strategy so far has the same fundamental limitation: chunks are encoded in isolation. A chunk that refers to "the president" doesn't know that chunk one defined "the president" as Biden. A chunk containing "this approach" doesn't know what approach was just described. The embedding captures what's in the chunk, but not where the chunk sits in the document.
Late chunking, developed by JinaAI, attacks this at the encoding level. Instead of chunking first and then encoding, you run the full document through the encoder first — getting context-aware token embeddings for every token, informed by the entire document. Then you pool those token embeddings into chunk representations after the fact. Because the token embeddings were computed with full-document context, the resulting chunk vectors carry that context with them. It's a promising direction, but it requires a model specifically trained for this approach — currently the JinaAI jina-embeddings-v3 family — so it's not yet universally available.
Contextual Retrieval, described by Anthropic in 2024, solves the same problem from the other side — through the text itself rather than the encoding. For each chunk, you prompt an LLM with the full document and the chunk together, asking it to generate a short description of how this chunk fits in the broader document. Something like: "This chunk describes the authentication flow for API tokens, which is step 3 of the 5-step onboarding process described in this guide." That description is prepended to the chunk before embedding. Now the embedding captures both the chunk's content and its document-level context.
Contextual Retrieval results: Anthropic reports that combining Contextual Retrieval with BM25 reduces the retrieval failure rate by approximately 67% compared to standard chunking. The cost is one LLM call per chunk at indexing time — but using prompt caching with a small model like Claude Haiku makes this economical. There's no added cost at query time. It's currently the best bang-for-buck retrieval improvement available without changing your retrieval architecture.
Chunking is a design decision, not a config value
The throughline across all six strategies is that there's no universal best chunking approach — there's only the right approach for your document type, query pattern, and indexing budget. Fixed-size chunking is a reasonable starting point for uniform prose. Semantic chunking is a low-cost upgrade for mixed-topic documents. Parent-child retrieval is the practical middle ground that most production systems eventually converge on. Contextual Retrieval is the highest-leverage improvement available if you can afford the indexing LLM calls.
RAPTOR and proposition-based chunking are more specialized: RAPTOR for corpora where broad thematic questions matter as much as specific lookups; propositions for systems where retrieval precision is the primary concern and indexing cost is secondary. Late chunking is one to watch — as model support broadens, it may become the cleanest general solution.
In the next lesson, we'll move beyond flat document retrieval entirely, and look at how knowledge graphs can answer the kinds of cross-document synthesis questions that no chunking strategy handles well on its own.