Esment Lab6 min read

What each stage of our memory retrieval cascade is worth

We switched off the graph, the reranker and both in Esment's retrieval cascade: what each did to recall, what the reranker cost in time, and what we changed.

The Esment Team

In short

  • On 150 questions over a reduced corpus, turning the graph off cost 8.7 points of full recall (92.7% to 84.0%); turning the reranker off cost 1.4 points (to 91.3%).
  • The reranker mattered more downstream. End to end, answer accuracy fell from 78.7% to 73.3% without it, though 150 questions could not rule out chance (p = 0.15).
  • At user scale the CPU reranker added about 3.5 seconds to the median search, almost the same at 2.4K and 10K memories, because its cost follows the candidate pool, not the store.
  • We kept the reranker and made it cheaper instead. These are Lab numbers from one machine: compare the configurations to each other, not to production.

A memory search is not one search. Ours is a cascade: several ways of finding candidates, then a second model that puts them in order. This is an ablation of that retrieval cascade: we turned the graph off, turned the reranker off, then turned off both, and measured what each stage was worth in recall and in time. It is a Lab note, so it includes what we got wrong and what we still don't know.

What the retrieval cascade does, in one picture§

A question goes through three searches at once: one for the exact words, one for meaning using embeddings, and one that walks the knowledge graph outward from the best hits. Their results are fused into a pool of 64 candidates. A reranker, a model that reads the question next to each candidate, then reorders the pool, and the top ten go to your assistant.

YourquestionfuseThreesearches64 candidatesRerankerreorderreads eachpairTop 10

Your question → Three searches → Reranker → Top 10

The October retrieval stack: fuse three searches into a pool, reorder it, keep ten.

The graph and the reranker are the two stages that cost the most to build and run, so they are the two we tested. How the graph is built is in our note on the temporal knowledge graph.

How we ran the retrieval ablation§

Before the numbers, the conditions, because they decide how to read them.

  1. Take 150 development questions from a public long-term-memory benchmark, LongMemEval, and keep only the memories that belong to them: 73,586 of 247,057.
  2. Switch off every background model, so only retrieval runs. A hosted model embeds the query.
  3. Run each question four ways: full stack, reranker off, graph off, both off. Ask for the top 10 sessions each time.
  4. Score each run: how often at least one right session is in the top 10 (recall any), how often every needed session is (recall all), and how early the first right one appears (MRR).

Two checks kept us honest. The full stack matched our reference run on 98.7% of retrieved sessions on average, so the setup was not broken. And because we cut the corpus down to those 150 questions, with no distractors from other cases, absolute recall here is not comparable to any other figure we have published.

The graph was worth more recall than the reranker§

Recall "any" barely moved: 99.3% for the full stack and 98.7% for every variant, a difference of one or two questions. Recall "all", which needs every session the answer depends on, is where the stages separate.

Full recall at top 10, by configuration

92.7%Full stack91.3%Reranker off84%Graph off80%Both off
Full recall at top 10, by configuration
Full stack92.7%
Reranker off91.3%
Graph off84%
Both off80%
Share of 150 questions where every needed session reached the top 10.Source: Esment Lab, October 2026

Compared question by question with the full stack, turning off the reranker lost 5 questions and won 3. With 150 questions that is not distinguishable from chance. Turning off the graph lost 14 and won 1; turning off both lost 21 and won 2.

The per-type results say where the graph helps. On multi-session questions, the ones that need facts from several conversations, full recall went from 82.4% (full stack) to 67.6% (graph off) and 52.9% (both off). On temporal-reasoning questions it went from 88.2% to 67.6% and 64.7%. On single-session questions all four configurations were at or near ceiling. A graph earns its place when the answer is spread across more than one place.

Ranking told a different story. The first right session appeared slightly earlier without the reranker (MRR 0.925 against 0.915), and the both-off run scored 0.922. If you only watched how early the first hit lands, you would conclude the reranker does nothing.

What the reranker bought: answers, not recall§

So we measured something closer to what you feel. We had a small open language model read the top ten results, answer each question, and a judge mark the answers, with the same prompt and settings in both arms.

Full stackReranker off
Answers judged correct78.7%73.3%
Questions only this arm got right168
Temporal reasoning91.2%79.4%
Multi-session64.7%55.9%
Knowledge update85.0%90.0%
Average prompt sizeabout 2,400 tokensabout 3,100 tokens

The gap is about five points, in favor of keeping the reranker. The paired test gives p = 0.15, so with 150 questions we cannot rule out chance, and knowledge-update questions went the other way. The prompt size is a clue to the mechanism: without the reranker, the top ten held longer sessions, so the model read about 700 more tokens and did worse with them.

The reader and judge are one small model, and the benchmark is English. The small open reranker we tested is English-only, and we have not measured this stage in other languages.

What the reranker cost at user scale§

The ablation corpus has 73,586 memories, more than most people will ever store. So we rebuilt it at two sizes closer to a real user: 2,405 and 10,193 memories. We ran 60 searches on each, 40 filtered to one question's memories and 20 unfiltered, with the reranker on and off. The reranker was the same small open model, on the CPU, over the 64-candidate pool.

Median search time, filtered queries

2.4K memories, rerankeroff315 ms2.4K memories, rerankeron3793 ms10K memories, rerankeroff560 ms10K memories, rerankeron4124 ms +3,564 ms
Median search time, filtered queries
2.4K memories, reranker off315 ms
2.4K memories, reranker on3793 ms
10K memories, reranker off560 ms
10K memories, reranker on4124 ms
The reranker added 3,478 ms at 2.4K memories and 3,564 ms at 10K, 92% and 86% of the search time.Source: Esment Lab, October 2026

Two things stand out. First, the reranker's cost barely changed with the size of the store: its work depends on the 64 candidates, not on how many memories exist. What grew with the store was everything else, from 315 to 560 ms. Second, the cost was about seven times the 500 ms budget we had set for a search.

This test has real limits. Only 5 and 21 distinct questions existed at the two sizes, so the 40 filtered queries repeated texts. We did not rule out query caching by measurement. The machine was swapping heavily, which inflates the tails, most of all for unfiltered queries on the larger set, where the noise swamped the reranker's share. Treat the absolute times as upper bounds and the ratios as the finding.

What we changed because of it§

The ablation said keep the reranker. The latency test said it was too slow to keep as it was. We made it cheaper rather than smaller, because shrinking the candidate pool to 32 or 16 cut time but also cut end-to-end accuracy.

  • Same engine for assistants. Along the way we found that the search tool assistants call skipped the vector stage and the reranker, so it was not the engine we had measured. We made it use the same path. Through that tool, accuracy on the same 150 questions went from 58.0% to 73.3%, matching the no-reranker result above.
  • A faster local reranker. We sorted candidates by length into batches and stopped requests from queuing behind each other. On the evaluation corpus, accuracy stayed at 78.7% with no discordant answers, the top ten was identical on 98.7% of questions, and the median reranker time fell from 2,053 to 1,176 ms.
  • A time limit. If the reranker fails or takes longer than 3 seconds, search returns the order from the fused pool instead of an error.
  • A cheaper cascade. Once the reranker was faster, the pool-of-64 cascade became the bottleneck on that corpus, at 7.9 seconds median. We fixed that too, down to 0.27 seconds, with accuracy unchanged.
  • One thing we did not ship by default. An 8-bit version of this English reranker was smaller but scored 75.3%, so we did not make it the default.

Those were measured on the evaluation corpus, not at the user-scale sizes above, so don't subtract them from the 3.5 seconds. They are what lies behind the faster, fully local search described in Esment 1.8.0.

What we still don't know§

  • Whether the reranker's gain is real at the size we measured. Five points with p = 0.15 is a lean, not a result.
  • How the reranker behaves outside English, which we tested in our multilingual memory search note. The benchmark and the reranker we tested are both English.
  • What search time looks like for real people. Our user-scale runs used a few repeated questions on a machine under pressure, and we haven't published a fresh run at those sizes since the changes.
  • Whether better retrieval is the main fix. When we read wrong answers, many had the evidence in view, as in why AI memory answers fail.
  • Whether benchmark questions look like yours. We wrote about that gap in the benchmark is not the territory.

That is the other half of why we publish the conditions. As of November 2026, retrieval in Esment is a stack we have taken apart, not just one we have assembled. If you are new to the vector stage that this stack starts from, what is an embedding is the place to begin, and how Esment works shows where search sits among the rest.

Questions, answered§

What is an ablation in AI memory retrieval?

An ablation turns one part of a system off and measures what changes. For a retrieval cascade that means running the same questions with the graph stage off, the reranker off, or both, then comparing recall and speed against the full setup.

Does a reranker improve recall?

In our test it barely moved recall, 92.7% against 91.3% on full recall. Its effect showed up later, in the answers a model wrote from the top ten results, where accuracy dropped about five points without it.

Why is a reranker slow?

It reads the question together with every candidate, one pair at a time, instead of comparing precomputed vectors. With 64 candidates on a CPU, that was about 3.5 seconds in our user-scale test, however large the memory store was.

Are these numbers what Esment users see?

No. They come from one development machine under memory pressure, on a corpus built from benchmark conversations. Only the differences between configurations carry meaning. The absolute times and scores are not production numbers.

Pass it on →

Keep reading

all posts →
Esment Lab3 min

What we learned building a temporal knowledge graph for AI memory

Why a flat list of facts fails as AI memory, how a temporal knowledge graph handles contradictions, and the four-stage retrieval cascade behind Esment.

#knowledgegraph #retrieval
Esment Lab7 min

Multilingual memory search: how we tested it and what we changed

Esment Lab note on multilingual memory search: how we built the test set, what a cross-language test showed, and what we changed. Limits included.

#multilingual #retrieval
Esment Lab8 min

Why AI memory answers fail: retrieval or reading?

We read the wrong answers from Esment's memory search. Most had the evidence in front of the model, which points at how memories get written, not found.

#memory #retrieval

a note for you

New posts, by email.

Fixes, guides and Lab notes on AI memory — about two a month. No product news, unsubscribe in one click.

Or follow the RSS feed