Esment Lab8 min read

Why AI memory answers fail: retrieval or reading?

We read the wrong answers from Esment's memory search. Most had the evidence in front of the model, which points at how memories get written, not found.

The Esment Team

In short

  • On 150 benchmark questions, 39 failed in at least one of two passes. In 31 of them (79%) every needed session was already in the top ten the model read.
  • Preference questions failed most (18 of 30), then multi-session (12 of 34). Most other types barely failed.
  • Turn dilution is a coverage problem at the level of single turns. About half of the top-ten slots were assistant replies, and the user's own turn with the fact was often missing.
  • What it means for writing memory: keep the user's turn, resolve dates when you write, and store preferences as general facts. Our fixes are proposed and simulated offline, not yet proven.

When a memory system gives a wrong answer, there are two places to look. Either the right memory never came back (a retrieval failure), or it came back and the model still got the answer wrong (a reading failure). The fix for each is different, so we sorted our own wrong answers into those two piles. This is why AI memory answers fail, as far as Esment's October search showed it, and what it changed about how we think memories should be written. It is a Lab note: we say what we measured, what we did about it, and what we don't know.

Why AI memory answers fail: retrieval or reading§

Retrieval puts a short list of memories in front of the model. Reading is everything after that: the model uses the list, or ignores it, or decides it says nothing relevant. For the longer version of how those two halves fit together, see our guide to persistent memory for AI agents. This post takes the next step: what happens when the list is right and the answer is still wrong.

The test set was the same 150 questions from the public LongMemEval benchmark that we used in our retrieval ablation. A small open language model read the top ten results and answered each question, twice, so we could tell stable failures from noise. For each failed answer we checked one thing: were the sessions that hold the answer already in the top ten?

Most failures had the evidence in front of the model§

Thirty-nine of the 150 questions failed in at least one of the two passes, and 26 failed in both. For the 39, we counted how much of the needed evidence had reached the reader.

Failed questions, by how much evidence the reader had

All needed sessionspresent31 questions 79% of the 39Some sessions missing7 questionsNo needed sessionpresent1 questions
Failed questions, by how much evidence the reader had
All needed sessions present31 questions
Some sessions missing7 questions
No needed session present1 questions
39 questions failed at least once over two passes. In 31, every session that holds the answer was in the top ten.Source: Esment Lab, October 2026

So 79% of the failures were reading or assembly failures, and only 8 were retrieval failures in the narrow sense (7 partial, 1 absent). That fits the overall numbers for the same setup: at least one right session reached the top ten for 99.3% of questions, and every needed session for 92.7%.

It also changed what we worked on. If retrieval is already mostly bringing the right sessions back, a better retriever has a small ceiling on this test. The bigger gap sits between the list and the answer.

The question types that fail most§

Failures were not spread evenly. They were concentrated in two question types.

Question typeQuestionsFailed at least onceFailed both passesEvidence status of those failures
Preference30181317 complete, 1 absent
Multi-session3412107 complete, 5 partial
Temporal34735 complete, 2 partial
Knowledge update20101 complete
Single-session, user fact18101 complete
Single-session, assistant fact1400none

Preference and multi-session questions made up 30 of the 39 failures. Two patterns stood out inside them.

Preference: the answer is in the context, the model abstains§

A preference question asks for advice that should bend around something the user said earlier. Take a synthetic version: Alex Rivera, a product designer in Lisbon, once mentioned buying a large power bank. Later Alex asks for tips to make a phone battery last longer. A good answer builds on the power bank.

The reader often did not. In the preference failures, the memory about the preference was usually already at rank 3 or better. What the model said instead fell into two groups: roughly half abstained ("none of the memories mention battery"), and half gave generic advice that ignored the personal detail. A regex count, which is only indicative, found a "the memories don't contain this" style answer in 58% of failed preference answers (18 of 31 answers) and in none of the correct ones.

In several cases the preference was expressed about a different topic from the question. A liking for a rooftop terrace, said in a conversation about another city, has to be carried over to a hotel question about Miami. A reader told to use only the memories, and to say when there's no information, reads that literally and says there isn't any.

Multi-session: a missing piece, or a count that comes up short§

Multi-session questions need facts from several conversations: how many, in total, how long. Two things went wrong. In some, one needed session was missing from the top ten, so a sum or a count could not be right. In others the evidence was all there and the model undercounted when it had to combine three or four sessions with states that changed, such as a subscription started and later cancelled. The reader was also too literal with arithmetic and inference, such as turning "7:00" and "two hours" into an arrival time.

Temporal failures were a different story. Four of the seven were noise: the same question passed on one pass and failed on the other, with the right dates in context. Across the whole 150, 13 questions flipped between passes, which is why we measure every proposed change on the specific questions it targets rather than on the overall score.

Turn dilution: the assistant's reply got stored, not the user's turn§

The first diagnosis counted sessions. A second analysis, on the full 500-question version of the benchmark and with four different readers, went down to single turns. In this benchmark setup, each stored item is a turn of a conversation. That is where something odd showed up.

The top ten holds ten turns. Across multi-session questions, 654 of 1,328 slots were assistant replies, and the top ten covered an average of only 3.6 distinct sessions. Among the turns that hold the answer, 260 of 327 were in context for multi-session questions and 33 of 44 for preference questions. The rest were absent even when their session was present.

What the top ten held–Assistant: a long reply–Assistant: more advice–Assistant: a summary–User: a different turnWhat the answer needed✓User: the turn with thefact✓Same session, one turnvs

What the top ten held: Assistant: a long reply; Assistant: more advice; Assistant: a summary; User: a different turn. What the answer needed: User: the turn with the fact; Same session, one turn.

Turn dilution: the right session is in the list, but the assistant's reply took the slot of the user's turn that holds the fact.

We call this turn dilution. The session is retrieved, so a session-level check says the evidence is present. But the sentence that actually states the fact, usually the user's own, was not selected, because the assistant's reply to it scored well too. In the benchmark's labels, 842 of 896 answer-bearing turns come from the user.

Counting turns this way, the recovery of evidence mattered more than any single reader. Missing a session or missing a turn together accounted for 15 to 18.5 failed multi-session questions (out of 133) for all four readers. The strongest reader nearly eliminated arithmetic errors, from 6.5 to 0.5 questions, but it did not reduce missing coverage. A stronger reader does not read a sentence that isn't there.

One more number: 22 questions failed in all eight passes (four readers, two passes each). For 19 of those 22, the cause was missing coverage, whether a session or a turn, not the reader.

What it means for how memory should be written§

Both diagnoses point at the write side and at how the list is assembled, more than at the search itself. We drew four conclusions. The first two are about assembly, the last two about writing. Only the first has been replayed offline; the other three are proposals we have not measured.

  1. Keep the user's turn with its session. Cap the turns taken from any one session, and add the user's own turns from the sessions already retrieved, ahead of the assistant's.
  2. Hand over preferences as a profile. Put them in a fixed block, and tell the reader to personalize with them, not to answer only from topic matches.
  3. Resolve dates when you write. Store "last Saturday" as a date, resolved against the day of the conversation, so the reader doesn't do the date arithmetic. Dated facts are also what let a temporal knowledge graph replace an old fact with a new one.
  4. Store general facts next to the original turns, not instead of them. Keep what was said, and add short facts about what the user said, with a link back to the turn.

For the first one, we replayed our stored top tens offline. Adding every user turn from the sessions already retrieved raised the number of multi-session questions with all answer turns in context from 91 to 114 (of 133), and the preference questions from 21 to 29 (of 30). The cost was about 3,600 more characters per query, roughly 1,000 tokens. A cheaper variant that added only the neighboring user and assistant turn reached 100 and 25.

Multi-session questions with every answer turn in context

91 of 133Top ten as served100 of 133Plus neighboring turn114 of 133Plus user turns ofretrieved sessions
Multi-session questions with every answer turn in context
Top ten as served91 of 133
Plus neighboring turn100 of 133
Plus user turns of retrieved sessions114 of 133
A replay of stored results, not a new run: it counts coverage of the turns that hold the answer, not answer accuracy.Source: Esment Lab, October 2026

The caveat matters. This measures coverage, not correct answers. It assumes the reader answers correctly once it has the evidence, and our own preference diagnosis says that is optimistic: in preference questions, the evidence was already there in most failures. So we treat the 114 as a ceiling.

The fourth idea is the one we are least sure of. A design for it extracts a short list of facts, preferences and events from each session, keeps the temporal phrase as written, and resolves it against the session date in plain code, not in the model. It never replaces the original turns. We know why: an earlier experiment that appended whole raw sessions helped where it brought missing evidence, 18.5 more questions right across 66 of them, and hurt where the evidence was already there, 10.5 fewer right across 428, with about 2,900 more tokens per question. New text is not free. The expected result of the facts design is about one point of accuracy, with a range from a small loss to 2.5 points gained. We put the chance that it clears our bar at about 30%, and we have not run it.

What we still don't know§

  • Whether the assembly fix holds up end to end. The replay counts turns in context. It does not show answers improving. We haven't re-served the questions with the change.
  • Whether a better retrieval depth would help. We only stored the top ten, so we could not measure a larger list or a per-session cap. A rough lexical stand-in suggested that about half of the 16 missing sessions would be recoverable.
  • Whether facts written at ingest help or hurt. The design is untested. An earlier attempt with a very small extraction model did not work, and a model that omits one instance can state a wrong count with more authority than the raw turn.
  • How much of this is the benchmark. The labels put nearly all answer-bearing turns on the user's side, and a real conversation may not. We wrote about that gap in the benchmark is not the territory, and about what any write-time transformation gives up in extraction is a bet.
  • How much was the judge or the label. A few failures were questions where the benchmark's own answer looked doubtful. We report them as such and did not count them as wins.

If you are building a memory of your own, the practical lesson is cheap to apply: when an answer is wrong, open the list the model was given before you tune the search. In our case, the answer was usually already there. The same logic applies to any assistant's prompt, as in context engineering.

Questions, answered§

Why do AI memory systems give wrong answers?

Two causes. Retrieval can fail to bring the right memory back, or the model can have the right memory in front of it and still misread it, abstain or answer generically. In our diagnosis of 39 failed questions, the second cause covered 31.

What is turn dilution in AI memory?

When memories are stored as individual conversation turns, the search can return the assistant's reply and miss the user's turn that holds the fact. All the right sessions are present, yet the sentence that answers the question is not. We call that turn dilution.

Which questions does AI memory get wrong most often?

In our test, preference questions, where the answer depends on a taste the user mentioned in another context, and multi-session questions, where the answer is spread across several conversations. Temporal questions failed less often, and some of those were noise.

Does a better retriever fix these failures?

Only some of them. Better retrieval would help the cases where a needed session or turn never arrived, but in our diagnosis most failures had the evidence present. Those need changes to how memories are written and handed to the reader.

Pass it on →

Keep reading

all posts →
Esment Lab6 min

What each stage of our memory retrieval cascade is worth

We switched off the graph, the reranker and both in Esment's retrieval cascade: what each did to recall, what the reranker cost in time, and what we changed.

#retrieval #evaluation
Esment Lab3 min

What we learned building a temporal knowledge graph for AI memory

Why a flat list of facts fails as AI memory, how a temporal knowledge graph handles contradictions, and the four-stage retrieval cascade behind Esment.

#knowledgegraph #retrieval
Agents & context7 min

Context engineering: what it is, where memory fits, and a practical checklist

Context engineering explained: where the term came from, the parts of a context, how it fails, where durable memory fits, and a checklist for ten minutes.

#contextengineering #AGENTS.md

a note for you

New posts, by email.

Fixes, guides and Lab notes on AI memory — about two a month. No product news, unsubscribe in one click.

Or follow the RSS feed