Every serious memory system for language agents is now reported against the same two or three numbers. Those numbers have gone up a lot in eighteen months. It is worth asking, carefully, what they have gone up at — because the benchmarks that produce them share a structural assumption that almost no deployed memory system satisfies: that the corpus holds still.
This piece contains no scores of our own. We think publishing another point on a saturated and largely non-comparable axis is not a contribution right now. Naming precisely what that axis fails to cover might be.
The corpus holds still#
The dominant conversational-memory benchmarks are question answering over a fixed transcript. A history is ingested once, the store is frozen, and questions are asked against it. Everything the evaluation observes happens on the read path.
A deployed memory system spends almost none of its life in that state. It is written to continuously. Facts arrive that contradict earlier facts. Users correct things. Records get deleted for reasons ranging from a typo to a legal request. Derived layers — summaries, entity graphs, embeddings — have to be kept consistent with a substrate that keeps moving. The interesting failures of a real memory system are mutation failures, and a frozen-corpus benchmark cannot express them.
This is worth stating precisely, because it is stronger than the usual complaint about benchmark coverage. The write path is not under-weighted in these evaluations. It is unscored by construction: the experimental design has no place to put it.
What recall measures#
Set the write path aside and look only at what is scored. Even there, the headline numbers turn out to measure something narrower than "does this system remember."
In LoCoMo, roughly 94% of the evaluation questions can be grounded in evidence from no more than two prior sessions, and the conversations run on the order of sixteen to twenty-six thousand tokens.Both properties are widely reported in the 2026 evaluation literature; see the survey Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers (arXiv:2603.07670) and Locomo-Plus (arXiv:2602.10715), which was built specifically to push past the original benchmark's factual-recall ceiling. That length fits inside a current context window. A meaningful share of the score is therefore reachable by having no memory system at all and simply passing the transcript — which makes the benchmark a weak instrument for the thing it is being used to justify.
The scoring conventions add their own distortion. Task-disclosed prompting and string-matching metrics conflate memory fidelity with prompt adaptation and generation style: a system can gain points by phrasing answers the way the metric likes, independently of whether it recalled anything better.
The sharpest evidence that recall and usefulness have come apart is MemoryArena, which embeds memory inside multi-session agentic tasks where later subtasks depend on what was learned earlier. Systems that score near-perfectly on LoCoMo fall to roughly 40–60% there.MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks (arXiv:2602.16313). The gap it reports is between passive recall and memory that has to inform a decision — the distinction this whole section turns on. A forty-point collapse when the same stored knowledge has to drive a decision rather than answer a quiz is not a tuning gap. It says the two tasks were never the same task.
The judge is inside the system#
There is a further problem that makes cross-vendor comparison unsound rather than merely noisy. In LLM-as-judge evaluation, the judge model and its prompt are components of the system under test. Changing either moves scores by several points. A lenient judge inflates everything uniformly, which is worse than it sounds: it also hides regressions, because the leniency absorbs them.
Once you add the other unpinned knobs — the answer model, the retrieval budget, whether the reader is told the question category, how the adversarial and unanswerable questions are handled — published gaps between competing systems largely reduce to harness differences. Independent re-runs of vendor claims have repeatedly landed well below the announced figures.Zep's re-examination of Mem0's state-of-the-art claim is the most-cited public instance, but the pattern is general: the reproduction is not usually contesting the implementation, it is contesting the harness.
The practical rule this implies is uncomfortable for marketing and obvious for research: a leaderboard delta smaller than the harness delta is not a result. Unless the judge, judge prompt, reader model, and token budget are pinned and published, a comparison between two systems is a comparison between two experimental setups.
The unscored properties#
If the frozen corpus hides the write path, what exactly is hidden? These are the questions we think a memory system should have to answer, none of which the standard evaluations ask:
- When a new fact contradicts an old one, does the old one actually stop being retrieved — and if it was wrongly superseded, can the loss be detected or reversed?
- When a record is deleted, does the deletion propagate to every derived layer that absorbed it: summaries, entity nodes, graph edges, vector indexes?
- Does retrieval precision hold up after months of accumulation, or does undiscarded stale material gradually poison it?
- Can each injected token be traced back to the utterance it came from, well enough to explain or repair a wrong memory?
- What does one remembered message cost, in latency and in inference, at ingest and at query?
- Under multi-tenancy, can one tenant's history influence another's retrieval?
Of these, only selective forgetting is explicitly tested anywhere in common use, and even there by a single benchmark.MemoryAgentBench is the usual exception cited. The 2026 literature on this is worth reading in full: Agentic Unlearning (arXiv:2602.17692), From Lossy to Verified: A Provenance-Aware Tiered Memory for Agents (arXiv:2602.17913), and the survey Toward Mnemonic Sovereignty (arXiv:2604.16548), which frames memory security across six lifecycle phases — write, store, retrieve, execute, share, and forget/rollback. Provenance is in a worse position still: current systems generally offer limited support for tracing where a memory item came from or whether it remains valid, and treating memory as provenance-bearing evidence is an acknowledged open problem rather than a solved feature. Several production memory layers delegate deletion and audit consistency to the calling application entirely.
Toward a better axis#
The field has started to notice. The 2026 crop of benchmarks is visibly trying to escape passive recall: MemoryArena putting memory inside dependent agentic tasks, StreamMemBench evaluating in a streaming rather than batch setting, AMA-Bench targeting long-horizon agentic applications, and work explicitly reframing the problem from recall toward forgetting.StreamMemBench (arXiv:2606.14571), AMA-Bench (arXiv:2602.22769), and From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents (arXiv:2604.20006). This is the right direction and it is moving fast.
What still has no benchmark is the axis we think matters most: correctness of the store as it changes. Not "can you find the answer in a fixed pile," but "after six months of writes, corrections, contradictions, and deletions, is the pile still true, still precise, and still explainable." Every property in the list above is a property of a system in motion, and every current evaluation observes the system at rest.
We are building toward measuring that, and we would rather say so plainly than post a number on the old axis. When we do publish results, they will come with the judge, the judge prompt, the reader model, the token budget, the seeds, and the per-category breakdown — because without those, as argued above, there is no result to report. An oracle-context ceiling belongs in that same disclosure, since it is the only clean way to separate a retrieval failure from a reader failure, and the two are routinely reported as one.
None of this says the existing benchmarks were a mistake. They gave the field a shared starting point, and a shared starting point is worth a great deal. But a map that was drawn for one purpose gets dangerous when it is the only map and everyone starts optimizing the map instead of the ground. The score is not the memory. The benchmark is not the territory.