---
title: Why AI memory answers fail: retrieval or reading?
description: We read the wrong answers from Esment's memory search. Most had the evidence in front of the model, which points at how memories get written, not found.
date: 2026-11-18
updated: 2026-11-18
author: The Esment Team
category: Esment Lab
tags: memory, retrieval, evaluation
canonical: https://esment.notas.ai/blog/why-ai-memory-answers-fail
---

# Why AI memory answers fail: retrieval or reading?

*The Esment Team · 2026-11-18 · [HTML version](https://esment.notas.ai/blog/why-ai-memory-answers-fail)*

## In short

- On 150 benchmark questions, 39 failed in at least one of two passes. In 31 of them (79%) every needed session was already in the top ten the model read.
- Preference questions failed most (18 of 30), then multi-session (12 of 34). Most other types barely failed.
- Turn dilution is a coverage problem at the level of single turns. About half of the top-ten slots were assistant replies, and the user's own turn with the fact was often missing.
- What it means for writing memory: keep the user's turn, resolve dates when you write, and store preferences as general facts. Our fixes are proposed and simulated offline, not yet proven.

When a memory system gives a wrong answer, there are two places to look. Either
the right memory never came back (a **retrieval** failure), or it came back and
the model still got the answer wrong (a **reading** failure). The fix for each
is different, so we sorted our own wrong answers into those two piles. This is
why **AI memory answers fail**, as far as Esment's October search showed it, and
what it changed about how we think memories should be written. It is a Lab
note: we say what we measured, what we did about it, and what we don't know.

## Why AI memory answers fail: retrieval or reading

Retrieval puts a short list of memories in front of the model. Reading is
everything after that: the model uses the list, or ignores it, or decides it
says nothing relevant. For the longer version of how those two halves fit
together, see our guide to
[persistent memory for AI agents](https://esment.notas.ai/learn/persistent-memory-for-ai-agents).
This post takes the next step: what happens when the list is right and the
answer is still wrong.

The test set was the same 150 questions from the public LongMemEval benchmark
that we used in
[our retrieval ablation](https://esment.notas.ai/blog/memory-retrieval-cascade-ablation). A small open
language model read the top ten results and answered each question, twice, so we
could tell stable failures from noise. For each failed answer we checked one
thing: **were the sessions that hold the answer already in the top ten?**

> [!NOTE]
> The evidence check in this diagnosis is at the level of whole sessions, not single turns. Turns come later in this post. The questions are synthetic benchmark conversations, in English, and the reader is one small model. Read the proportions, not the absolute scores.

## Most failures had the evidence in front of the model

Thirty-nine of the 150 questions failed in at least one of the two passes, and
26 failed in both. For the 39, we counted how much of the needed evidence had
reached the reader.

**Figure: Failed questions, by how much evidence the reader had.** 39 questions failed at least once over two passes. In 31, every session that holds the answer was in the top ten.

| Item | Value (questions) |
|---|---|
| All needed sessions present | 31 |
| Some sessions missing | 7 |
| No needed session present | 1 |

Source: Esment Lab, October 2026
So **79% of the failures were reading or assembly failures**, and only 8 were
retrieval failures in the narrow sense (7 partial, 1 absent). That fits the
overall numbers for the same setup: at least one right session reached the top
ten for 99.3% of questions, and every needed session for 92.7%.

It also changed what we worked on. If retrieval is already mostly bringing the
right sessions back, a better retriever has a small ceiling on this test. The
bigger gap sits between the list and the answer.

## The question types that fail most

Failures were not spread evenly. They were concentrated in two question types.

| Question type | Questions | Failed at least once | Failed both passes | Evidence status of those failures |
|---|---|---|---|---|
| Preference | 30 | 18 | 13 | 17 complete, 1 absent |
| Multi-session | 34 | 12 | 10 | 7 complete, 5 partial |
| Temporal | 34 | 7 | 3 | 5 complete, 2 partial |
| Knowledge update | 20 | 1 | 0 | 1 complete |
| Single-session, user fact | 18 | 1 | 0 | 1 complete |
| Single-session, assistant fact | 14 | 0 | 0 | none |

Preference and multi-session questions made up 30 of the 39 failures. Two
patterns stood out inside them.

### Preference: the answer is in the context, the model abstains

A preference question asks for advice that should bend around something the user
said earlier. Take a synthetic version: Alex Rivera, a product designer in
Lisbon, once mentioned buying a large power bank. Later Alex asks for tips to
make a phone battery last longer. A good answer builds on the power bank.

The reader often did not. In the preference failures, the memory about the
preference was usually already at rank 3 or better. What the model said instead
fell into two groups: roughly half **abstained** ("none of the memories mention
battery"), and half gave **generic advice** that ignored the personal detail.
A regex count, which is only indicative, found a "the memories don't contain
this" style answer in 58% of failed preference answers (18 of 31 answers) and in none of
the correct ones.

In several cases the preference was expressed about a different topic from the
question. A liking for a rooftop terrace, said in a conversation about another
city, has to be carried over to a hotel question about Miami. A reader told to
use only the memories, and to say when there's no information, reads that
literally and says there isn't any.

### Multi-session: a missing piece, or a count that comes up short

Multi-session questions need facts from several conversations: how many, in
total, how long. Two things went wrong. In some, one needed session was missing
from the top ten, so a sum or a count could not be right. In others the
evidence was all there and the model **undercounted** when it had to combine
three or four sessions with states that changed, such as a subscription started
and later cancelled. The reader was also too literal with arithmetic and
inference, such as turning "7:00" and "two hours" into an arrival time.

Temporal failures were a different story. Four of the seven were noise: the
same question passed on one pass and failed on the other, with the right dates
in context. Across the whole 150, 13 questions flipped between passes, which is
why we measure every proposed change on the specific questions it targets rather
than on the overall score.

## Turn dilution: the assistant's reply got stored, not the user's turn

The first diagnosis counted sessions. A second analysis, on the full
500-question version of the benchmark and with four different readers, went down
to single turns. In this benchmark setup, each stored item is a turn of a conversation.
That is where something odd showed up.

The top ten holds ten turns. Across multi-session questions, **654 of 1,328
slots were assistant replies**, and the top ten covered an average of only 3.6
distinct sessions. Among the turns that hold the answer, 260 of 327 were in
context for multi-session questions and 33 of 44 for preference questions. The
rest were absent even when their session was present.

**Figure.** Turn dilution: the right session is in the list, but the assistant's reply took the slot of the user's turn that holds the fact.

| What the top ten held | What the answer needed |
|---|---|
| Assistant: a long reply | User: the turn with the fact |
| Assistant: more advice | Same session, one turn |
| Assistant: a summary |  |
| User: a different turn |  |
We call this **turn dilution**. The session is retrieved, so a session-level
check says the evidence is present. But the sentence that actually states the
fact, usually the user's own, was not selected, because the assistant's reply
to it scored well too. In the benchmark's labels, 842 of 896 answer-bearing
turns come from the user.

Counting turns this way, the recovery of evidence mattered more than any single
reader. Missing a session or missing a turn together accounted for 15 to 18.5
failed multi-session questions (out of 133) for all four readers. The strongest
reader nearly eliminated arithmetic errors, from 6.5 to 0.5 questions, but it
did not reduce missing coverage. A stronger reader does not read a sentence that
isn't there.

One more number: 22 questions failed in all eight passes (four readers, two
passes each). For 19 of those 22, the cause was missing coverage, whether a
session or a turn, not the reader.

## What it means for how memory should be written

Both diagnoses point at the write side and at how the list is assembled, more
than at the search itself. We drew four conclusions. The first two are about
assembly, the last two about writing. Only the first has been replayed offline;
the other three are proposals we have not measured.

> [!STEPS]
> 1. **Keep the user's turn with its session.** Cap the turns taken from any one session, and add the user's own turns from the sessions already retrieved, ahead of the assistant's.
> 2. **Hand over preferences as a profile.** Put them in a fixed block, and tell the reader to personalize with them, not to answer only from topic matches.
> 3. **Resolve dates when you write.** Store "last Saturday" as a date, resolved against the day of the conversation, so the reader doesn't do the date arithmetic. Dated facts are also what let [a temporal knowledge graph](https://esment.notas.ai/blog/temporal-knowledge-graph-ai-memory) replace an old fact with a new one.
> 4. **Store general facts next to the original turns, not instead of them.** Keep what was said, and add short facts about what the user said, with a link back to the turn.

For the first one, we replayed our stored top tens offline. Adding every user
turn from the sessions already retrieved raised the number of multi-session
questions with all answer turns in context from 91 to 114 (of 133), and the
preference questions from 21 to 29 (of 30). The cost was about 3,600 more
characters per query, roughly 1,000 tokens. A cheaper variant that added only
the neighboring user and assistant turn reached 100 and 25.

**Figure: Multi-session questions with every answer turn in context.** A replay of stored results, not a new run: it counts coverage of the turns that hold the answer, not answer accuracy.

| Item | Value (of 133) |
|---|---|
| Top ten as served | 91 |
| Plus neighboring turn | 100 |
| Plus user turns of retrieved sessions | 114 |

Source: Esment Lab, October 2026
The caveat matters. This measures **coverage**, not correct answers. It assumes
the reader answers correctly once it has the evidence, and our own preference
diagnosis says that is optimistic: in preference questions, the evidence was
already there in most failures. So we treat the 114 as a ceiling.

The fourth idea is the one we are least sure of. A design for it extracts a
short list of facts, preferences and events from each session, keeps the
temporal phrase as written, and resolves it against the session date in plain
code, not in the model. It never replaces the original turns. We know why:
an earlier experiment that appended whole raw sessions helped where it brought
missing evidence, 18.5 more questions right across 66 of them, and hurt where
the evidence was already there, 10.5 fewer right across 428, with about 2,900
more tokens per question. New text is not free. The expected result of the
facts design is about one point of accuracy, with a range from a small loss to
2.5 points gained. We put the chance that it clears our bar at about 30%, and
we have not run it.

> [!ESMENT]
> Esment saves memories as separate entries with a date, a kind and a source, and builds a short brief before a chat starts, as described in [how Esment works](https://esment.notas.ai/learn/how-esment-works). These findings are about the next layer: which entries and which turns go into that list, and how preferences and dates are written. As of November 2026, the changes in this post are proposals backed by offline replays, not shipped behavior.

## What we still don't know

- **Whether the assembly fix holds up end to end.** The replay counts turns in context. It does not show answers improving. We haven't re-served the questions with the change.
- **Whether a better retrieval depth would help.** We only stored the top ten, so we could not measure a larger list or a per-session cap. A rough lexical stand-in suggested that about half of the 16 missing sessions would be recoverable.
- **Whether facts written at ingest help or hurt.** The design is untested. An earlier attempt with a very small extraction model did not work, and a model that omits one instance can state a wrong count with more authority than the raw turn.
- **How much of this is the benchmark.** The labels put nearly all answer-bearing turns on the user's side, and a real conversation may not. We wrote about that gap in [the benchmark is not the territory](https://esment.notas.ai/research/the-benchmark-is-not-the-territory), and about what any write-time transformation gives up in [extraction is a bet](https://esment.notas.ai/research/extraction-is-a-bet).
- **How much was the judge or the label.** A few failures were questions where the benchmark's own answer looked doubtful. We report them as such and did not count them as wins.

If you are building a memory of your own, the practical lesson is cheap to
apply: when an answer is wrong, open the list the model was given before you
tune the search. In our case, the answer was usually already there. The same logic applies to any assistant's prompt, as in [context engineering](https://esment.notas.ai/blog/context-engineering).

## Questions, answered

### Why do AI memory systems give wrong answers?

Two causes. Retrieval can fail to bring the right memory back, or the model can have the right memory in front of it and still misread it, abstain or answer generically. In our diagnosis of 39 failed questions, the second cause covered 31.

### What is turn dilution in AI memory?

When memories are stored as individual conversation turns, the search can return the assistant's reply and miss the user's turn that holds the fact. All the right sessions are present, yet the sentence that answers the question is not. We call that turn dilution.

### Which questions does AI memory get wrong most often?

In our test, preference questions, where the answer depends on a taste the user mentioned in another context, and multi-session questions, where the answer is spread across several conversations. Temporal questions failed less often, and some of those were noise.

### Does a better retriever fix these failures?

Only some of them. Better retrieval would help the cases where a needed session or turn never arrived, but in our diagnosis most failures had the evidence present. Those need changes to how memories are written and handed to the reader.
