Memory & Learning Shipped
Five files asking whether the planned memory layer was too simple — and finding that the storage layer was over-provisioned while the recall layer was missing entirely.
The question
How do you build a layer on top of frozen models that stores what agents learn and lets a future session act as if the model had learned it? And specifically: is the plan — an MCP server to the backend — naive? The verdict is stated on the first page rather than buried: not naive, aimed at the wrong bottleneck.
What it found
- The storage already existed and was already good — a three-tier memory system with supersede-not-overwrite lifecycle and a real vector index. What did not exist was ingestion or automatic recall: memories exist only when a caller explicitly writes them, and the documentation store had zero automatic recall at all.
- Six disconnected memory systems across the author's projects, never reconciled with each other.
- The single most decision-relevant number in the pack: in Vercel's internal evaluation, model-judgment retrieval was never invoked in 56% of eval cases, while a compressed always-loaded 8KB index hit 100%. Deterministic injection beats asking the model to decide it needs to remember.
- Naive similarity search serves superseded values 15–40% of the time — which is the entire argument for validity ranges rather than deletion.
- A bare filesystem-with-grep beat a purpose-built memory product on the LoCoMo benchmark (74.0% vs a reported 68.5%), and a competing "100%" claim turned out to be a model reading full sessions with retrieval bypassed. Vendor memory benchmarks are close to unusable.
- Letting an LLM continuously consolidate its own memories is actively dangerous: in one study it caused a model to fail 54% of problems it had previously solved unaided. In another, 18,282 tokens of accumulated knowledge collapsed to 122 tokens after a monolithic rewrite, with accuracy dropping from 66.7% to 57.1%.
- Memory poisoning is worse than prompt injection in kind, not degree: one bad write persists and compounds.
Retrieval that actually fires
Vercel's internal evaluation: model-judgment retrieval was never invoked at all in 56% of cases. This is the argument for deterministic injection over asking a model to notice that it has something to remember — and the reason the memory work in flight is about injection rather than about better search.
What letting a model curate its own memory cost
Accuracy fell from 66.7% to 57.1% alongside it, and in a separate study continuous self-consolidation caused a model to fail 54% of problems it had previously solved unaided. Compression that looks like tidying is how a memory layer deletes the thing it was built to keep.
What it changed
It produced a design and a plan, both committed, and the design's problem statement reproduces the 56% and 15–40% figures directly. None of the planned artifacts have been built. That is the honest status, and it is why Memory carries a preview marker on this site: the storage and recall layer is genuinely live in production, and the part that puts a memory in front of a running session is not.
Sources
- Packer et al. — MemGPTThe three-tier architecture the existing storage layer already resembles.
- Rasmussen et al. — Zep: a temporal knowledge graph for agent memoryBi-temporal semantics, borrowed rather than adopted wholesale.
- Wu et al. — LongMemEval
- Shinn et al. — ReflexionVerbal reinforcement beats blind retry — but the buffer resets across tasks.
- Wang et al. — Voyager
- Sumers et al. — Cognitive Architectures for Language AgentsThe episodic / semantic split.
- Liu et al. — Lost in the MiddleWhy an always-loaded compressed index beats a long context.
Also cited in the pack, without links
- Agentic Context Engineering (the 18,282→122 token collapse) and MemOS — arXiv preprints named in the files.
- "Useful Memories Become Faulty When Continuously Updated by LLMs" — the 54% regression result.
- Practitioner sources: Anthropic's memory-tool documentation, claude-mem, claude-self-reflect, Karpathy's LLM-wiki pattern, and Steve Yegge's account of rejecting markdown and vector memory after burning 350k lines on a failed project.