The conversation that starts over every time
Ask a RAG pipeline what's in your knowledge base and it'll answer in milliseconds — that's the whole point of building one. Ask a long-running agent what a user told it yesterday, and most of the time it draws a blank, not because the information wasn't valuable, but because nothing in the system was built to hold onto it once the session closed.
That gap doesn't show up in demos, where every conversation starts fresh and stays short. It shows up the moment an agent has to persist — a coding assistant working the same codebase across weeks of sessions, a support agent that's supposed to remember a customer's history, a personal assistant that's supposed to know your preferences without being told twice. In all three cases, users notice memory by its absence: the agent re-asks a question it was already answered, contradicts a preference it agreed to last week, treats every session like the first one.
It's tempting to assume retrieval already solves this — you have a RAG pipeline, why not just index the conversation transcripts too? But retrieval and memory are answering different questions. Retrieval answers "what do I need to know to respond to this query," pulled from a corpus someone else curated. Memory answers "what have I learned about this specific relationship, over time" — and critically, the agent itself is the one writing it, turn by turn, as the relationship unfolds. That's a different engineering problem, with a different failure mode, and it starts with deciding what actually deserves to survive past the session that produced it.
Two clocks running at different speeds
Every agent already has one kind of memory, whether you designed for it or not: the context window. Everything said in the current session lives there, available to reason over, and gone the moment the session ends unless something outside that window captured it first. Call this working memory — fast, complete within its scope, and by default, completely disposable.
Long-term memory is a deliberate second system, sitting outside the context window, that survives the session boundary on purpose. Nothing puts information there automatically — an explicit write step has to decide a fact is worth persisting and commit it to a durable store the agent can query in a future, unrelated session. That write step is the entire discipline; everything else in this lesson is about how to do it well.
Long-term memory itself splits again, along a line borrowed from how we already talk about human memory. Episodic memory holds specific remembered events — in session twelve, the user asked you to avoid using a particular dependency. Semantic memory holds what got generalized out of those events — after enough sessions where that preference held, it becomes "this user avoids third-party dependencies," a standing fact the agent can apply proactively in session forty, without needing to recall session twelve at all.
That distillation step — turning a pile of specific episodes into a smaller number of durable, generalized facts — is what makes long-term memory usable at scale instead of just an ever-growing transcript archive. But it raises the next question immediately: what does the store that holds those facts actually look like?
The catch: Storing raw transcripts is complete but expensive and noisy to retrieve from — you're searching conversation, not facts. Storing only the distilled, semantic version is cheap and relevant, but lossy: extraction can misread a one-off remark as a standing preference, or flatten nuance the original exchange had. Most production systems end up needing both — a full record for auditing, and a distilled layer for fast, everyday retrieval.
Two ways to build the store
One architecture family builds long-term memory the way it builds everything else: with a vector store. An LLM pass reads the conversation, extracts candidate memory statements, embeds them, and writes them into an index. Retrieval at query time is semantic search over that index — the same read pattern RAG already uses, except the corpus is one the agent wrote about you, instead of one an engineer curated from documentation.
Mem0 is the most visible example of this pattern, and by its own count in a 2026 state-of-the-field survey, this has become a genuinely crowded category — 21 frameworks, 20 vector stores, three different hosting models. That's a sign of real demand, but also a sign the space hasn't consolidated around one obvious right answer yet.
A second architecture family takes a structurally different approach: the temporal knowledge graph, the pattern behind Zep's memory layer. Facts are modeled as edges between entities in a graph — rather than "the user prefers X" sitting alone as a vector, it's a relationship between a user-entity and a preference-entity — and every edge carries a validity interval, a record of when that fact became true and, if it later changed, when it stopped being true. When a new fact contradicts an old one, the graph doesn't overwrite anything; it closes the old edge's interval and opens a new one, which means the system can answer not just "what does the user prefer now" but "what did the user prefer as of three months ago," and reason across relationships between entities, not just similarity between texts.
Which one you reach for depends on what your facts actually look like. Loosely structured personal preferences with no strong relational structure fit the extraction-plus-vector pattern fine, and it's the simpler of the two to stand up. Facts about entities whose state changes over time — and where silently overwriting a contradiction would be a real loss, not just noise — are where the temporal graph earns its extra complexity. Either way, the store you choose only matters if it's actually cheaper to use than the alternative it's replacing, which is where the economics come in.
The catch: This is still a fast-moving, largely vendor-defined space, not a settled discipline the way RAG architecture has become. Treat "which memory architecture is best" as an open, actively-contested question for now — pick based on what your facts look like, not based on which vendor's benchmark post is newest.
The bill for a conversation that never forgets
The naive way to give an agent continuity across a long conversation is to just keep replaying the whole transcript into context on every turn. It works, in the sense that nothing gets lost — and it gets linearly more expensive with every exchange, because you're paying to re-read the entire history every single time, whether or not most of it is relevant to the current turn.
Selective memory retrieval is the alternative: instead of replaying everything, pull back a small number of condensed, relevant facts and leave the rest of the history where it is. The retrieval stays bounded no matter how long the relationship with the user has gone on, because you're querying a distilled store, not replaying a growing log.
Mem0 puts numbers on that gap in its own LoCoMo long-conversation benchmark results: its memory system scored 92.5 while using roughly 6,956 tokens per query, against roughly 26,000 tokens for a naive full-context approach on the same benchmark. Zep separately claims its temporal-graph approach beats baseline retrieval by 18.5% on long-horizon accuracy while cutting latency by nearly 90%. Directionally, both point the same way — selective retrieval over a distilled store scales better than replaying everything — and that's a token-cost argument as much as a design one, the same accounting discipline you'd apply to any other line item in an LLM system's bill.
Getting the economics right doesn't end at query time, either — a memory store that only ever grows has its own cost, which is where lifecycle management comes in.
The catch: These are vendor-published figures, not independently verified. Treat the mechanism — bounded retrieval structurally beats unbounded replay — as the durable lesson, and the specific percentages as claims to re-check before repeating them as fact in anything meant to stay accurate for a while.
Memory that never gets edited is not memory, it's a log
Writing new memories is the easy part — extraction runs continuously as new episodes happen, and each one becomes a candidate fact. What separates a real memory system from a growing pile of extracted notes is what happens after the write: whether anything ever revisits, merges, or retires what's already there.
Consolidation is that revisiting step. New facts get checked against existing ones, near-duplicates get merged, and contradictions get resolved instead of both versions sitting in the store at once. This is exactly where the two architectures from earlier behave differently in practice: a temporal graph resolves a contradiction structurally, by closing the old edge's validity interval and opening a new one, so the history stays intact and queryable. A vector store has no equivalent built-in mechanism — without an explicit update-or-merge step, the old and new facts both stay retrievable, and a query can just as easily surface the stale one as the current one.
Forgetting is the other half of the lifecycle, and it's a designed capability, not a failure to prevent. It serves two different purposes that are easy to conflate. Staleness pruning is about retrieval quality — old, no-longer-relevant facts dilute the store and make it easier for a query to surface something out of date. Compliance deletion is a different obligation entirely — a user asking to be forgotten needs the fact actually removed, not just down-weighted in a ranking, and a memory architecture that can't distinguish "less relevant" from "must not exist anymore" isn't going to satisfy that request.
The catch: Consolidate too aggressively and you risk merging two facts that were actually distinct, or losing nuance a blunt summarization step can't preserve. Consolidate too little and the store grows into an ever-larger, self-contradicting pile that retrieval can't cleanly disambiguate. There's no default setting that's safe for every domain — it has to be tuned against what a wrong merge actually costs you.
A third leg, not a rebrand of retrieval
Line memory up against the two systems you already know how to build, and the distinction holds up cleanly. Prompting is what you hand the model this turn. Retrieval is what the model pulls from a corpus someone else curated, scoped to a single query. Memory is what the agent itself wrote, about a specific relationship, meant to outlive the session that produced it — and unlike the other two, it has to handle contradiction and time, not just similarity, because the facts it holds are allowed to change.
None of this is settled the way RAG architecture is settled. The tooling landscape is genuinely crowded and still consolidating, the standout benchmark numbers are vendor-published rather than independently adjudicated, and the honest read on today's evidence is "real gap, real early tooling" rather than "solved problem, pick any framework." That's not a reason to skip it — the underlying need, an agent that doesn't start over every session, is only going to matter more as agents get put on longer-running work. It's a reason to choose an architecture based on what your facts actually look like — loose, low-structure preferences toward extraction and a vector store; entities whose relationships change over time, where a silent overwrite would actually cost you something, toward a temporal graph — rather than picking whichever vendor published the newest benchmark post.
That's the discipline underneath everything in this lesson: memory isn't a feature you bolt onto an agent once retrieval and prompting are working. It's a third system, with its own write path, its own lifecycle, and its own economics — and treating it with the same rigor you'd apply to any other piece of production infrastructure is what turns "the agent remembers things" from a demo trick into something you can actually rely on.