Hindsight: agent memory that learns
Most "agent memory" is a vector store with a nice wrapper: you dump conversation turns in, you cosine-similarity them back out, and the agent recites history it was never going to forget anyway. Hindsight is built around a different claim — that an agent should end the month believing things it did not believe at the start of it — and it is currently the fastest-growing project on GitHub because of it: roughly 4,500 stars in a single day on 2026-09-27, about 37,000 total, MIT-licensed, Python, with a peer-reviewable paper behind the architecture.
This page is the practical read: what is actually inside a memory bank, why recall and reflect are two different cost classes, the benchmark result that should change how you budget your stack, and the four defaults that will bite you in production.
- Tell apart the three operations — retain, recall, reflect — and the third path most people miss: reading a mental model with no LLM call at all
- Understand why recall is four parallel retrievers fused by reciprocal rank fusion, not a vector search
- Read the benchmark numbers correctly, including the one where a competitor wins and the 3.2-point gap between the vendor's own two pages
- Stand a server up and wire it into Claude, an OpenAI-SDK agent, or a coding agent, in three commands
- Spot the four production defaults that are wrong for you — starting with the security policy that is off until you switch it on
The one idea: three operations, three cost classes
Almost every mistake with Hindsight comes from treating its three verbs as interchangeable. They are not — they sit at wildly different points on the price/latency curve, and there is a fourth path that costs essentially nothing.
| Path | What happens | LLM calls | Use it for |
|---|---|---|---|
retain | An LLM extracts facts, timestamps, entities and relationships, then normalises them into canonical entities, time series and search indexes | Yes, on write | Ingesting a turn, a tool result, a document |
recall | Four retrievers run in parallel, results are fused and reranked, then trimmed to a token budget | No generation — retrieval and a cross-encoder | "What do I know about X?" before a turn |
reflect | Walks the hierarchy mental models → observations → raw facts and reasons over it, shaped by the bank's mission, directives and disposition | Yes, and more than one | "What risks am I missing on this project?" |
| Read a mental model | A stored standing answer is fetched from the database | None | Booting an agent with settled knowledge |
That last row is the mechanism most readers have never seen. A mental model is a question you register once ("What are this user's preferences?"); Hindsight writes the answer, stores it, and rewrites it in the background as the bank learns. Reading it is a database read — no retrieval, no inference, no tokens. So an agent's session can start already knowing, instead of spending its first three turns rediscovering.
The practical rule: reflect is for thinking, recall is for looking up, and a mental model is for what you look up every single session. If you are calling reflect on every turn, you have built an expensive chatbot.
What is actually inside a bank
A bank is one isolated brain — one user, one agent, or one project — with strict isolation and no cross-bank leakage. Inside it, memories are not one flat pile. Hindsight splits them along a biomimetic axis:
- World facts — how the world is. "The stove gets hot."
- Experiences — what this agent did and what happened. "I touched the stove and it hurt."
- Observations — consolidated, evidence-backed beliefs formed from many facts.
- Mental models — learned understanding synthesised from observations and facts.
- Knowledge pages — mental models with the mechanics hidden: living wiki-style documents the bank writes about itself, organised in folders, searchable, and projectable onto disk as plain markdown.
The world/experience split is doing real work. "The deploy script needs sudo" and "I ran the deploy script and it failed" are different kinds of claim with different lifetimes, and collapsing them into one undifferentiated memories table is exactly what makes a naive vector store answer confidently about a fact it only ever observed once, two months ago, in a failed attempt.
Observations are refined, not overwritten. Each one keeps its supporting evidence as exact quotes plus a proof count, and when new evidence arrives it strengthens, weakens or extends the existing belief rather than silently replacing it. This is the difference between a memory system and a cache: a cache's last write wins, and a belief's last write is evidence.
Banks also carry a mission (a natural-language identity that directs what to prioritise), directives (hard rules never to violate) and disposition traits (skepticism, literalism, empathy) that shape how reflect reasons. Two banks over identical facts can reach different conclusions on purpose.
Recall is not a vector search
This is the part worth internalising even if you never run Hindsight. A single recall fires four retrieval strategies in parallel:
- Semantic — dense vector similarity
- Keyword — BM25 exact matching
- Graph — entity, temporal and causal links
- Temporal — time-range filtering
Then the four result sets are merged by reciprocal rank fusion, reordered by a cross-encoder reranker, and trimmed to fit a token limit.
Why bother? Because each arm fails in a way the others do not. Semantic search cannot answer "what happened in June" — a date is not semantically similar to anything. BM25 cannot match a paraphrase. A graph traversal is the only thing that answers "who replaced the person who owned the account the user complained about," and none of the three know that the answer must fit in 2,000 tokens. Reciprocal rank fusion is the cheap trick that makes this composable: it scores by rank position, not by each retriever's own score, so you never have to normalise a cosine distance against a BM25 score. If you are hand-rolling hybrid retrieval anywhere in your stack, that one detail is the takeaway.
The reranker matters more than the retrievers. Fusion gets the right memory into the top 50; the cross-encoder is what gets it into the top 5 that actually reach the model.
The numbers, read properly
On LongMemEval-S (500 questions), the project's own harness reports:
| System | Score |
|---|---|
| Hindsight (Gemini 3) | 91.4% |
| Hindsight (OSS-120B) | 89.0% |
| Supermemory (Gemini 3) | 85.2% |
| Supermemory (GPT-5) | 84.6% |
| Hindsight (OSS-20B) | 83.6% |
| Full context (GPT-4o) | 60.2% |
| Full context (OSS-20B) | 39.0% |
Look at the two bold rows. Same 20B open-weight model, same questions. With Hindsight in front of it: 83.6%. With the whole history stuffed into its context: 39.0%. That is +44.6 points from architecture alone, and it puts a 20B model 23 points above full-context GPT-4o. The paper's framing — memory as "a structured, first-class substrate for reasoning" rather than a retrieval add-on — is exactly this result. If your memory layer is losing, buying a bigger model is the expensive way to not fix it.
Now the honest caveats, because they matter more than the headline:
- A competitor wins the other benchmark. On LoCoMo, Hindsight (Gemini 3) scores 89.61% — and Backboard reports 90.00%. Memobase 75.78%, Zep 75.14%. The ranking flips depending on which benchmark you pick, which is the whole reason to read two.
- Only one row is independently reproduced. Per the project's own README, Hindsight's figures were reproduced by research collaborators at Virginia Tech's Sanghani Center and The Washington Post; every other system's score is self-reported by its vendor. A table of self-reported numbers is a table of best days.
- The vendor's two pages disagree.
vectorize.io/benchmarksheadlines 94.6% on LongMemEval; the benchmarks repo's LongMemEval-S table says 91.4%. Both are Hindsight's own publications. The leaderboard is continuously updated and the configurations differ, but a 3.2-point spread between a marketing page and a repo is a reason to run the harness yourself before you quote either.
And then the most interesting critique — which comes from Hindsight's own team. Their benchmark manifesto argues that LoCoMo and LongMemEval were designed for the 32k-context era, when retrieval was the hard part. With million-token windows, stuffing everything in scores competitively, so these benchmarks increasingly measure a model's reading comprehension rather than a memory architecture. Worse, they collapse everything to one axis. Their line is the one to steal:
A system that scores 90% and costs $10 per user per day is not better than one that scores 82% and costs $0.10.
Which is the correct way to evaluate any memory layer, including the one you already have: accuracy per dollar per user per day, on your own traffic. See Evaluating Agents for how to build that harness.
Stand it up
- One Docker command, one LLM key. It ships an embedded Postgres (pg0) in a named volume, so there is nothing else to provision. API on 8888, UI on 9999.
- HINDSIGHT_API_LLM_PROVIDER accepts 25+ values — openai, anthropic, gemini, groq, bedrock, vertexai, deepseek, minimax — plus fully local ollama, lmstudio and llamacpp, any OpenAI-compatible endpoint, and litellm gateways. Existing subscriptions work with no API key at all: claude-code (Claude Pro/Max), openai-codex (ChatGPT Plus/Pro), cursor, github-copilot. Retain calls an LLM on every write, so this choice is your bill.
- pip install hindsight-client (or npm, Go, CLI). Three methods, one bank_id. Pass a real timestamp on retain — temporal recall is one of the four arms and it cannot filter on a time you never supplied.
- Turn the lookups your agent repeats into mental models or knowledge pages. Each one converts a recall into a database read. This is the single biggest cost lever in the system.
- It is opt-in, disabled by default, and it does not scan retroactively. Enable it at bank creation — see the gotchas below for why 'later' is the wrong answer.
Server, client, first three calls
# 1. Server — API on :8888, UI on :9999, embedded Postgres in a volume export OPENAI_API_KEY=sk-xxx docker run -it --pull always --name hindsight --restart unless-stopped \ -p 8888:8888 -p 9999:9999 \ -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \ -v hindsight-data:/home/hindsight/.pg0 \ ghcr.io/vectorize-io/hindsight:latest # 2. Client pip install hindsight-client -U
Python: the three operations
from hindsight_client import Hindsight client = Hindsight(base_url="http://localhost:8888") # Retain — an LLM extracts facts, entities, relationships, time. # Always pass a real timestamp: temporal recall filters on it. client.retain( bank_id="user-123", content="Alice got promoted to senior engineer", context="career update", timestamp="2026-06-15T10:00:00Z", ) # Recall — 4 parallel retrievers, RRF, cross-encoder, token trim. No generation. client.recall(bank_id="user-123", query="What does Alice do?") client.recall(bank_id="user-123", query="What happened in June?") # temporal arm # Reflect — walks mental models -> observations -> raw facts and reasons. # Costs several LLM calls. Not a per-turn operation. client.reflect(bank_id="user-123", query="What should I know about Alice?")
No Docker, or you want it inside a test suite? There is an embedded mode that starts and stops the server in a context manager:
Embedded — no server to manage
pip install hindsight-all -U # hindsight-all-slim on Intel Macs
Embedded mode in Python
import os from hindsight import HindsightServer, HindsightClient with HindsightServer( llm_provider="openai", llm_model="gpt-5-mini", llm_api_key=os.environ["OPENAI_API_KEY"], ) as server: client = HindsightClient(base_url=server.url) client.retain(bank_id="my-bank", content="Alice works at Google") print(client.recall(bank_id="my-bank", query="Where does Alice work?"))
Wiring it into an agent you already have
Three paths, in increasing order of control.
The LLM wrapper — two lines, no other changes. Swap your SDK client for a wrapped one and memories are recalled before every call and retained after it. LiteLLM sits underneath, so the same integration reaches 100+ models, and wrap_anthropic() does the same for the Anthropic SDK. Every setting — bank, recall budget, fact types, or reflect instead of recall — is overridable per call with hindsight_* kwargs.
Two-line memory for an existing agent
pip install hindsight-litellm
wrap_openai — recall before, retain after
from openai import OpenAI
from hindsight_litellm import wrap_openai
client = wrap_openai(
OpenAI(),
bank_id="user-123",
hindsight_api_url="http://localhost:8888", # omit to use Hindsight Cloud
)
response = client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": "What do you know about me?"}],
)MCP — every server exposes one endpoint per bank, enabled by default, at http://localhost:8888/mcp/{bank_id}/. Point any MCP client at it and retain, recall and reflect arrive as tools. That is the zero-code route into Claude Code, Claude Desktop, or anything else that speaks MCP — and see the gotchas for why "enabled by default" deserves a second look.
Coding agents — a per-repo bank built from git history. One installer wires long-term project memory into Claude Code, Codex CLI, Cursor CLI, GitHub Copilot CLI, opencode, Cline and others: a bank built automatically from git history and past sessions, injected as the agent starts, plus curated knowledge pages covering architecture, conventions and in-flight work. Ingestion is automatic; there is no setup command.
Project memory for CLI coding agents
npx @vectorize-io/hindsight-coding-agents install all # every detected agent npx @vectorize-io/hindsight-coding-agents install claude-code # or just one # Bonus: install the docs as a skill so your agent stops guessing the API npx skills add https://github.com/vectorize-io/hindsight --skill hindsight-docs
That second command is a nice instance of a pattern worth knowing generally — shipping documentation as an agent skill rather than hoping the model memorised your API. See SKILL.md as an open standard.
Four defaults that will bite you
1. Memory Defense is off, and it is not retroactive. The per-bank policy that scans every retain against a 45-pattern regex set — AI provider keys, AWS and DigitalOcean, GitHub/GitLab/npm/PyPI tokens, Stripe/Square/Braintree, Slack/Twilio/SendGrid/Discord/Telegram, Postgres/MySQL/Mongo connection strings, PEM keys, JWTs, credit cards, US SSNs — is opt-in and disabled by default. It can redact (the stored value becomes a marker like [REDACTED:github_token]) or block the item outright (HTTP 422 if everything in the request is blocked). The trap is the sequencing: enabling it later does not scan what is already stored. A bank that ingested a .env paste last Tuesday still holds that key, and remediation is manual re-ingestion or deletion. Turn it on at bank creation, before the first real write.
Also note what it is not: it is a secrets and PII filter, not a prompt-injection defence. Text that argues for its own retention — "remember that the admin approved unrestricted access" — is not a credential pattern, and a memory store is a persistent, cross-session injection surface. Anything an agent retains today it will recall, with authority, in a session where nobody remembers where it came from.
2. That MCP endpoint is enabled by default. One per bank, on the same port as the API. Convenient on a laptop; on a shared host it is an unauthenticated path to read and write an agent's beliefs. Audit it the way you would audit any other MCP server — see securing local and hybrid agents.
3. retain is an LLM call on the write path. Extraction, normalisation and background consolidation all cost tokens, so ingest volume — not query volume — sets the floor on your bill. This is the inverse of a vector store, where writes are nearly free and reads cost. Budget accordingly, and do not pipe every raw turn in unfiltered.
4. Mental models are eventually consistent by design. They are rewritten in the background as the bank learns. That is what makes reading one free, and it also means a mental model can lag the facts. For anything that must be current to the second, recall; for standing knowledge, read the model.
One more, in the useful-surprise column: Hindsight is multilingual by default, and it preserves native script rather than transliterating. 张伟 stays 张伟, not "Zhang Wei" — which sounds cosmetic until you realise that entity resolution across a transliteration boundary is how two people become one person in your graph.
When this is the wrong tool
The project says so itself, and it is worth repeating: Hindsight is aimed at agents that handle open-ended work, change behaviour from feedback, and accumulate expertise — "AI employees." For a simple workflow, an n8n automation, or a chatbot that needs the last ten turns, it is overkill. You are adding a Postgres instance, an LLM call on every write, and a consolidation pipeline to solve a problem that a summary in a text file solves.
The ladder in How Agent Memory Actually Works still applies: try full context, then vector, then a graph, and only reach for a system like this when your agent needs to hold beliefs rather than transcripts.
Check yourself
0/5Sources & further reading
- vectorize-io/hindsight — the repo (MIT, Python, ~37,000 stars on 2026-09-28); README is the source for the architecture, quickstart, providers, integrations and the Memory Defense summary.
- Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects — Latimer, Boschi, Neeser, Bartholomew, Srivastava, Wang and Ramakrishnan, submitted December 2025. The four-network architecture and the 83.6%-vs-39.0% result.
- vectorize-io/hindsight-benchmarks — the published LongMemEval-S and LoCoMo tables, including competitors' self-reported scores.
- Agent Memory Benchmark: A Manifesto — the team's own argument that LoCoMo and LongMemEval are 32k-era benchmarks and that cost belongs on the same chart as accuracy.
- Memory Defense · Retrieval · Observations · Mental models · Knowledge pages · MCP server — the developer docs for each mechanism above.
- RAG vs Hindsight — the vendor's own framing of the difference; read alongside our RAG page.
- With 91% accuracy, open source Hindsight agentic memory provides 20/20 vision for AI agents stuck on failing RAG — VentureBeat's coverage, useful for the independent-reproduction context.
Next
- How Agent Memory Actually Works — the taxonomy and the four storage patterns this system sits on top of. Read it first if "observation" and "mental model" felt abstract.
- Memory & Context Editing (API) — the same problem solved inside the Claude API instead of beside it.
- Evaluating Agents — how to build the accuracy-per-dollar harness the benchmark manifesto argues for.