When your documents aren't made of words
Every retrieval system we've discussed so far starts with the same assumption: your content is text, and your queries are text. That assumption is almost always false in practice. Product catalogs contain images. Internal knowledge bases hold scanned PDFs with complex layouts. Research archives store charts and diagrams that don't survive OCR. The moment you hit any of these, your text pipeline silently fails — and you're usually not told why.
Multimodal retrieval is the family of techniques for breaking that assumption. Instead of requiring everything to be converted into text before it can be searched, these approaches let you retrieve across modalities directly — matching a text query to an image, or understanding a PDF page as a visual object rather than a stream of extracted characters.
There are three distinct approaches to know. Two are production-ready today; one is still emerging. The clearest way to understand them is in order of how they handle the gap between modalities.
One embedding space for text and images
The simplest multimodal retrieval approach is also the most elegant: train a model so that text and images share the same embedding space. When you embed the sentence "a golden retriever running on a beach," the resulting vector should end up near the embedding of an actual photo of that scene. This is what CLIP does — and what SigLIP, ALIGN, and their successors do as well.
The practical implication is that natural language image search works without any intermediate steps. You don't need to tag images, write captions, or maintain a separate keyword index. You embed your image corpus once, embed the user's query, and run a standard ANN search across both. A user searching for "damaged product packaging" retrieves the images that match — even if those images have never been labeled with those exact words.
This pattern is especially well-suited to product catalogs, medical imaging, and design asset libraries — anywhere the visual content carries meaning that text descriptions would either miss or require significant human annotation to capture.
The catch: CLIP-style retrieval works on image-to-text matching, but it doesn't understand document layout. A scanned PDF page is technically an image, but a CLIP model will try to match it semantically as a scene rather than parse its structure, tables, and headings. For that problem, you need a different approach entirely.
Reading documents without reading text
Traditional PDF retrieval pipelines have a fragility problem that practitioners know well but rarely talk about. The standard approach — extract text with OCR, chunk it, embed it — fails quietly in ways that are hard to diagnose. OCR breaks on multi-column layouts. Tables get flattened into garbage. Charts become nothing. Footnotes end up concatenated with body text. By the time you're doing retrieval, you've lost the structure that made the document meaningful in the first place.
ColPali takes a completely different approach: skip the text extraction entirely, and treat each document page as an image. Using a vision-language model — specifically PaliGemma, a model jointly trained on vision and language — ColPali embeds patches of the page image using late interaction. Instead of producing one vector per page, it produces a grid of patch embeddings, and relevance is scored by matching query token embeddings against all the page patches.
If that sounds familiar, it should — it's the same late interaction idea as ColBERT, which we covered in the retrieval architectures lesson, applied to visual tokens instead of text tokens. The model isn't reading the text; it's reading the page the same way a person reads it, taking in visual structure, spatial relationships, and visual emphasis all at once.
The results on document understanding benchmarks like DocVQA are striking. ColPali consistently outperforms text-extraction pipelines on documents with complex layouts — not because the underlying task changed, but because it stopped throwing away information before retrieval even began. Vespa and Qdrant both have production-ready integrations.
The catch: ColPali is computationally heavier than text retrieval — storing and searching patch embeddings per page costs more than a single dense vector. And it requires a GPU-capable serving environment for the vision model. For clean, text-heavy PDFs where OCR works reliably, the overhead may not be worth it. The value shows up sharply when documents have layout complexity that text extraction mangles.
The frontier that isn't ready yet
Audio and video retrieval follow the same conceptual logic — embed the modality, search the embedding space — but the infrastructure hasn't converged the way it has for images and documents. Models like ImageBind and similar multimodal encoders can jointly embed audio, video, text, and images into a shared space, but their performance and tooling support in production RAG stacks are still research-grade.
The use cases are real. Customer support call libraries, surveillance systems, and media archives all contain retrieval problems that are fundamentally audio or video-native. But today, most production systems that need to retrieve audio or video still rely on a text intermediary — transcription for audio, frame extraction plus captioning for video — rather than direct dense retrieval on the modality itself.
Current state: Unless you're building something highly specialized, treat dense audio/video retrieval as a research direction to watch rather than a production tool to adopt today. The transcription-first approach is slower to build but significantly more reliable on current models and tooling.
The right tool depends on what your content actually is
Multimodal retrieval isn't one thing — it's a spectrum of approaches matched to specific content types. CLIP-style embeddings are the right choice when your corpus contains images that need to be searched semantically. ColPali is the right choice when your documents have layout complexity that text extraction can't preserve. Dense audio and video retrieval is worth monitoring but not yet worth building on in most production environments.
The pattern that tends to surprise people is ColPali. If you've ever inherited a PDF-heavy knowledge base and felt the quiet pain of an OCR pipeline that silently degrades your retrieval quality, ColPali is the specific solution for that exact problem — and it works today, without requiring you to re-annotate or re-structure your document corpus.
In the next lesson, we'll go one level deeper into the infrastructure that powers all the retrieval we've discussed — looking at how vector indexes actually work, from HNSW to IVF-PQ, and what the parameter choices mean for your recall and latency tradeoffs.