Your filter and your index are fighting each other
Most RAG systems eventually hit a requirement that sounds simple: only retrieve documents that belong to this user, or only search within this department's knowledge base, or only return results published after this date. Metadata filters. How hard could it be?
Hard enough that it's one of the most common failure points in enterprise RAG. The problem isn't adding a filter — it's that the way you apply that filter can silently destroy your retrieval quality, and most systems default to an approach that does exactly that.
There are three distinct strategies for combining metadata filtering with vector search, and only one of them actually works well at scale. By the end of this lesson you'll know the difference, and you'll know how to make your retrieval system understand both what the user is asking and what constraints apply — simultaneously.
Shrinking the haystack shrinks your chances
The intuitive approach is pre-filtering: apply the metadata constraint first to get a subset of documents, then run your ANN search only within that subset. It feels right — you're working with a smaller, already-relevant pool.
The problem is that ANN indexes are built and calibrated on the full corpus. HNSW, for example, builds its graph of short- and long-range edges assuming all the vectors are present. When you suddenly restrict the search to a small subset — say, the 2,000 documents belonging to one tenant in a multi-tenant system — the graph structure no longer makes sense for that subset. The traversal paths that lead to approximate nearest neighbors in the full graph may not exist within your filtered slice.
The result is poor recall. You asked for the ten most relevant documents from this department, and your vector index confidently returned ten documents that aren't actually the best matches because the ANN search degraded badly on a small filtered population.
The failure mode: The smaller the filtered subset relative to the full index, the worse your ANN accuracy becomes. Pre-filtering looks safe but is the most likely strategy to silently return wrong results without raising any errors.
Retrieve everything, keep almost nothing
Post-filtering flips the order: run ANN search globally first to find the top-K nearest neighbors, then filter that result set by metadata. This preserves ANN accuracy — you're always searching the full, well-connected graph — but it trades one problem for another.
If your filter is selective enough, you might retrieve 100 candidates and keep only 3 after filtering. You asked for the top 10 results, and you got 3. The system has to decide: re-query with a larger K? Re-query until enough results pass the filter? Or just return 3 and pretend that's fine?
Each of those options is awkward. Re-querying is wasteful and makes latency unpredictable. Setting K very large upfront wastes compute on candidates that will be thrown away. And returning fewer results than requested is a silent quality problem — the user sees a short list and doesn't know why.
The failure mode: Post-filtering's cost scales inversely with filter selectivity. A filter that matches 50% of the corpus is fine. A filter that matches 5% means you're doing 20× as much work — or getting incomplete results.
Evaluate the filter during traversal, not around it
The right solution is to make the filter a first-class participant in graph traversal, not a step that happens before or after it. This is what ACORN and filtered HNSW variants do: metadata predicates are evaluated inline as the graph search explores candidate nodes.
Instead of restricting which nodes are visible before search begins, or pruning results after the fact, the traversal simply skips nodes that don't match the filter as it moves through the graph. The graph structure remains intact for the full corpus. The ANN algorithm can still use its long-range edges to navigate efficiently. But it only returns nodes that satisfy the predicate.
This eliminates both pre-filter's accuracy degradation and post-filter's wastefulness. The search does roughly the same amount of work regardless of filter selectivity, and it returns exactly the number of results you asked for — all of which satisfy the metadata constraint.
Both Qdrant and Weaviate implement variants of this approach. If you're choosing a vector store for a system that will need metadata filtering, this capability should be on your checklist.
Let the model extract the filter from the question
Even with a solid filtered vector search backend, there's still a user experience problem: your users are typing natural language, not structured queries. "Find papers about transformers published after 2022 with more than 100 citations" contains both a semantic search intent and three metadata constraints. Separating those by hand, for every possible query shape, is brittle and doesn't scale.
The self-querying retriever pattern delegates this parsing to an LLM. You give the model a schema describing your document metadata — what fields exist, what types they are, what values are valid — and a natural language query. The model returns both a semantic query string and a structured filter object. Those get passed to your filtered vector search backend simultaneously.
So "papers about transformers from after 2022 with more than 100 citations" becomes a semantic query for "transformer architecture" paired with filters {year: {$gt: 2022}, citations: {$gt: 100}}. The user didn't have to know those fields existed. The model figured it out.
LangChain's Self-Query Retriever implements this pattern — you define the metadata schema once, it handles the parsing. The quality of filter extraction depends on how clearly your schema is described and how unambiguous the user's intent is, but for well-structured enterprise data it works reliably.
Filtering is an architectural decision, not an afterthought
The lesson here is that metadata filtering isn't a feature you bolt on after you've built your retrieval pipeline — it's a constraint that shapes your index architecture. Pre-filtering degrades ANN quality on small subsets. Post-filtering wastes compute and produces unpredictable result counts. Inline filtering during graph traversal is the approach that scales correctly, and it requires vector store support to do it right.
Self-querying sits above all of this: once your backend can filter correctly, you can let an LLM extract structured constraints from natural language so users never need to know the underlying schema exists.
Together these patterns cover the full stack of metadata-aware retrieval — from how the index evaluates predicates all the way up to how the user's intent gets translated into those predicates in the first place.
In the next lesson, we'll zoom out to production pipeline patterns — how teams wire retrieval, fusion, and reranking together end-to-end, and what the most common architectures look like in real systems.