How Agent Memory Actually Works
Ask a chatbot the same question twice in two sessions and it answers like a stranger both times. That is not a bug — it is the default. A raw language model has no memory between calls. Everything it "remembers" inside one conversation lives in the context window, and when that conversation ends, it is gone.
Memory is the machinery you bolt on to fix that: the systems that decide what an agent should carry forward, where to store it, and how to pull the right piece back at the right moment. In 2026 this stopped being a side quest and became a first-class part of agent design, with its own benchmarks, frameworks, and a genuine research literature. This page is the map.
- Understand why a context window is not memory — and where the boundary actually is
- Tell apart the four memory types agents use: working, episodic, semantic, procedural
- Compare the four storage patterns: full-context, vector/RAG, knowledge-graph, and compaction/summarization
- See how Claude, ChatGPT, and Gemini each implement memory today
- Choose a memory approach for your own agent without over-engineering it
The one idea to hold onto: context ≠ memory
The most common confusion is treating a big context window as "memory." It isn't. The context window is working space for one turn — it is refilled from scratch on every call, it is finite, and it is expensive. Attention also degrades across it (the "lost in the middle" effect covered in Context Engineering).
Memory is different in three ways:
| Context window | Memory | |
|---|---|---|
| Lifespan | One request | Across sessions, days, forever |
| Size | Fixed token ceiling | Effectively unbounded (external store) |
| Cost | Paid every single turn | Paid once to write; cheap to reference |
| Access | Everything, always in view | Selective — retrieve only what's relevant |
The whole game of agent memory is moving the right information between these two: write durable facts out of the window so you don't pay for them every turn, and pull them back in only when this specific step needs them. Get that flow right and an agent can operate for weeks on a context window that only ever holds a few thousand relevant tokens.
The four kinds of memory
Borrowing (loosely) from cognitive science, the 2026 agent ecosystem has converged on four categories. You rarely need all four — but naming them stops you from building one blob that does everything badly.
- What's in the context window right now — the current task, the last few turns, the tool results from this step. Volatile by design. This is the scratchpad, not the archive. Managing it well is context engineering; it is not persistence.
- Specific things that happened, with a timestamp. 'On Tuesday the user said checkout was broken; on Wednesday support marked it resolved.' Episodic memory is inherently temporal — the order and the when matter. It's what lets an agent reason about a history rather than a snapshot.
- Durable facts and preferences, stripped of when you learned them. 'The user prefers metric units.' 'This customer is on the enterprise plan.' Semantic memory is largely atemporal — it represents what the agent believes is currently true, not the event where it found out.
- How to do something — reusable skills, workflows, and learned routines. The least mature of the four in practice. In tools like Claude Code this often lives as instruction files (CLAUDE.md) and reusable skills rather than an automatic store.
A useful test: if you would answer with "when did that happen?" it's episodic; if you'd answer with "what's true?" it's semantic; if you'd answer with "here's how" it's procedural; and if it only matters for the next few seconds, it's working memory and doesn't need persisting at all.
The four storage patterns
Once you know what to remember, you pick how to store and retrieve it. There are four dominant patterns, roughly in order of complexity. Most real systems combine two or three.
1. Full-context (stuff it all in)
Keep the entire history and re-send it every turn. Zero infrastructure, perfect recall — until you hit the token ceiling, the cost curve, or "lost in the middle." Fine for short assistants; a dead end for anything long-running. This is the baseline every other pattern improves on.
2. Vector / RAG memory
Write each memory as an embedding into a vector database; at query time, embed the current turn and retrieve the top-k most similar memories. This is retrieval-augmented generation pointed at conversation history instead of documents. Cheap, scalable, and the default for semantic recall of facts and preferences.
Its weakness: similarity ≠ relevance for temporal or multi-hop questions. "What did we decide after the budget got cut?" is an ordering question, and cosine similarity has no sense of time or of chaining two facts together.
3. Knowledge-graph memory
Store memories as entities and relationships — nodes and edges, often with timestamps on the edges. To answer a question you traverse the graph rather than fuzzy-matching vectors. This is what makes multi-hop and temporal reasoning tractable ("who replaced the person who owned the account the user complained about?"). Frameworks like Zep/Graphiti built their whole pitch around temporal knowledge graphs. The cost is real engineering: extraction, entity resolution, and keeping the graph from rotting.
4. Compaction & summarization
Periodically compress the running history into a distilled summary and continue from that — trading verbatim recall for a smaller, cheaper window. This is what /compact does in Claude Code, and what "auto-summary" does in many chat products. It's the cheapest form of long-term memory and often the first one you actually need. Its risk: the summary silently drops the one detail you needed. See Long-Running Agent Harnesses for how this plays out over hours-long runs.
Real systems layer these. A common 2026 stack: compaction for the running conversation, vector for semantic facts, and a graph on top only when temporal/multi-hop queries actually show up in your traffic. Don't build the graph until you feel the pain the graph solves.
How the big three do it
Every major assistant now ships some memory. They are not the same thing, and the differences matter.
| Product | What it remembers | How it works (roughly) |
|---|---|---|
| Claude | Two layers: an app-level memory of your preferences, and a developer-facing memory tool for agents. | The Claude app memory stores facts across chats; the API memory tool plus context editing lets an agent write notes to a client-side store and auto-prune stale tool results to survive long runs. |
| ChatGPT | "Saved memories" (explicit facts) plus reference to your past chats. | A mix of user-stated facts and automatically extracted preferences, injected into the system context on later turns. User-editable and toggleable. |
| Gemini | Personal context drawn from your chats and, optionally, the wider Google account surface. | Recalls details from previous conversations and can personalize using account context, subject to your privacy controls. |
Two takeaways. First, consumer memory is mostly semantic — preferences and facts — not full episodic replay. Second, if you're building an agent, the product's built-in memory is not your memory system; you own that layer, using primitives like Claude's memory tool or an external framework.
Turn a raw model into a note-taking agent (the cheapest real memory)
You have a file called MEMORY.md that persists between our sessions. At the END of each session, append any durable facts worth keeping: - my stable preferences (tools, formats, style) - decisions we made and WHY - open threads to resume next time At the START of each session, read MEMORY.md first and use it. Keep it under 30 lines — when it grows past that, consolidate and delete anything stale. Never store secrets or credentials.
That single pattern — write durable notes to an external file, read them back next time — is the 80/20 of agent memory. Most of the framework machinery below is a more automatic, more scalable version of exactly this.
Measuring memory: the LoCoMo benchmark
You can't improve what you can't measure, and memory was hard to measure until benchmarks arrived. The most cited is LoCoMo ("Evaluating Very Long-Term Conversational Memory of LLM Agents"): very long multi-session conversations — hundreds of turns across dozens of sessions — with question-answer pairs in five flavors: single-hop, multi-hop (cross-session), temporal reasoning, open-domain, and adversarial.
What LoCoMo reveals is the pattern to design around: systems do fine on single-hop factual recall and fall apart on temporal and multi-hop questions. That failure mode is exactly why knowledge-graph memory exists — it's the pattern that lifts those two categories most. When you evaluate your own agent's memory, weight the multi-hop and temporal cases heavily; single-hop recall flatters almost everything.
Choosing an approach without over-building
- Do nothing. Working memory (the context window) is enough. Adding a memory store here is pure overhead.
- Start with note-taking to an external file, or the product's built-in memory. This covers most 'remember my preferences' needs at near-zero cost.
- Add compaction/summarization. Keep the load-bearing facts, drop the play-by-play. This is where long-running agents live.
- Add vector/RAG memory. Retrieve the top-k relevant memories per turn instead of re-sending everything.
- Only now reach for a knowledge graph — or a managed framework (Mem0, Letta, Zep, LangMem) that gives you one without hand-rolling extraction and entity resolution.
The trap is starting at step five. Graph memory is impressive in demos and expensive in production. Climb the ladder; stop at the first rung that solves your actual problem.
Check yourself
0/4The bottom line
Memory is not one feature you turn on — it's a flow you design: what leaves the window, where it's stored, and how it comes back. Name the four memory types so you don't build one blob for all of them. Start at the cheapest storage pattern that solves your problem and climb only when you feel the pain of the next one. And measure with temporal and multi-hop cases, because single-hop recall makes everything look smarter than it is.
Memory is the other half of Context Engineering: context engineering decides what fills the window this turn; memory decides what survives between turns. Together they're what separates a chatbot from an agent that gets better the longer you work with it.
Sources & further reading
- Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo) — the canonical benchmark; project page.
- Effective context engineering for AI agents — Anthropic on compaction, note-taking, and just-in-time retrieval.
- Claude memory tool & context editing docs — the developer-facing primitive.
- Agent Memory Techniques — 30 runnable notebooks covering conversation buffers, vector stores, knowledge graphs, episodic/semantic memory, Mem0, Letta, Zep, Graphiti, and LoCoMo.
- The State of AI Agent Memory 2026 — vendor report on memory architectures and benchmarks (read with the usual self-reported-numbers caution).
- Related on AILmanac: Context Engineering · Long-Running Agent Harnesses · RAG · Claude app memory · Memory & Context Editing (API).