---
title: What each stage of our memory retrieval cascade is worth
description: We switched off the graph, the reranker and both in Esment's retrieval cascade: what each did to recall, what the reranker cost in time, and what we changed.
date: 2026-11-18
updated: 2026-11-18
author: The Esment Team
category: Esment Lab
tags: retrieval, evaluation, latency, knowledge-graph
canonical: https://esment.notas.ai/blog/memory-retrieval-cascade-ablation
---

# What each stage of our memory retrieval cascade is worth

*The Esment Team · 2026-11-18 · [HTML version](https://esment.notas.ai/blog/memory-retrieval-cascade-ablation)*

## In short

- On 150 questions over a reduced corpus, turning the graph off cost 8.7 points of full recall (92.7% to 84.0%); turning the reranker off cost 1.4 points (to 91.3%).
- The reranker mattered more downstream. End to end, answer accuracy fell from 78.7% to 73.3% without it, though 150 questions could not rule out chance (p = 0.15).
- At user scale the CPU reranker added about 3.5 seconds to the median search, almost the same at 2.4K and 10K memories, because its cost follows the candidate pool, not the store.
- We kept the reranker and made it cheaper instead. These are Lab numbers from one machine: compare the configurations to each other, not to production.

A memory search is not one search. Ours is a cascade: several ways of
finding candidates, then a second model that puts them in order. This is an
**ablation** of that retrieval cascade: we turned the graph off, turned the
reranker off, then turned off both, and measured what each stage was worth
in recall and in time. It is a Lab note, so it includes what we got wrong
and what we still don't know.

## What the retrieval cascade does, in one picture

A question goes through three searches at once: one for the exact words, one
for meaning using [embeddings](https://esment.notas.ai/learn/what-is-an-embedding), and one that
walks the [knowledge graph](https://esment.notas.ai/docs/memories/graph) outward from the best hits.
Their results are fused into a pool of 64 candidates. A reranker, a model that
reads the question next to each candidate, then reorders the pool, and the top
ten go to your assistant.

**Figure.** The October retrieval stack: fuse three searches into a pool, reorder it, keep ten.

Your question → Three searches → Reranker → Top 10
The graph and the reranker are the two stages that cost the most to build and
run, so they are the two we tested. How the graph is built is in
[our note on the temporal knowledge graph](https://esment.notas.ai/blog/temporal-knowledge-graph-ai-memory).

## How we ran the retrieval ablation

Before the numbers, the conditions, because they decide how to read them.

> [!STEPS]
> 1. Take 150 development questions from a public long-term-memory benchmark, LongMemEval, and keep only the memories that belong to them: 73,586 of 247,057.
> 2. Switch off every background model, so only retrieval runs. A hosted model embeds the query.
> 3. Run each question four ways: full stack, reranker off, graph off, both off. Ask for the top 10 sessions each time.
> 4. Score each run: how often at least one right session is in the top 10 (recall any), how often every needed session is (recall all), and how early the first right one appears (MRR).

Two checks kept us honest. The full stack matched our reference run on 98.7%
of retrieved sessions on average, so the setup was not broken. And because we
cut the corpus down to those 150 questions, with no distractors from other
cases, **absolute recall here is not comparable** to any other figure we have
published.

> [!WARNING]
> Absolute numbers on this page are not production numbers. They come from one development machine under heavy memory pressure, on a corpus built from benchmark conversations. The differences between configurations are what count.

## The graph was worth more recall than the reranker

Recall "any" barely moved: 99.3% for the full stack and 98.7% for every
variant, a difference of one or two questions. Recall "all", which needs every
session the answer depends on, is where the stages separate.

**Figure: Full recall at top 10, by configuration.** Share of 150 questions where every needed session reached the top 10.

| Item | Value (%) |
|---|---|
| Full stack | 92.7 |
| Reranker off | 91.3 |
| Graph off | 84 |
| Both off | 80 |

Source: Esment Lab, October 2026
Compared question by question with the full stack, turning off the reranker
lost 5 questions and won 3. With 150 questions that is not distinguishable
from chance. Turning off the graph lost 14 and won 1; turning off both lost
21 and won 2.

The per-type results say where the graph helps. On multi-session questions,
the ones that need facts from several conversations, full recall went from
82.4% (full stack) to 67.6% (graph off) and 52.9% (both off). On
temporal-reasoning questions it went from 88.2% to 67.6% and 64.7%. On
single-session questions all four configurations were at or near ceiling. A
graph earns its place when the answer is spread across more than one place.

Ranking told a different story. The first right session appeared slightly
earlier without the reranker (MRR 0.925 against 0.915), and the both-off run
scored 0.922. If you only watched how early the first hit lands, you would
conclude the reranker does nothing.

## What the reranker bought: answers, not recall

So we measured something closer to what you feel. We had a small open
language model read the top ten results, answer each question, and a judge
mark the answers, with the same prompt and settings in both arms.

| | Full stack | Reranker off |
|---|---|---|
| Answers judged correct | 78.7% | 73.3% |
| Questions only this arm got right | 16 | 8 |
| Temporal reasoning | 91.2% | 79.4% |
| Multi-session | 64.7% | 55.9% |
| Knowledge update | 85.0% | 90.0% |
| Average prompt size | about 2,400 tokens | about 3,100 tokens |

The gap is about five points, in favor of keeping the reranker. The paired
test gives p = 0.15, so with 150 questions **we cannot rule out chance**, and
knowledge-update questions went the other way. The prompt size is a clue to
the mechanism: without the reranker, the top ten held longer sessions, so the
model read about 700 more tokens and did worse with them.

> [!NOTE]
> We fixed the decision rule before looking: switch the reranker off only if answers lose no more than 2 points and the loss is not significant. The loss was larger than 2 points, so the rule said keep it, even though the test itself lacked the power to settle the question.

The reader and judge are one small model, and the benchmark is English. The
small open reranker we tested is English-only, and we have not measured this
stage in other languages.

## What the reranker cost at user scale

The ablation corpus has 73,586 memories, more than most people will ever
store. So we rebuilt it at two sizes closer to a real user: 2,405 and 10,193
memories. We ran 60 searches on each, 40 filtered to one question's memories
and 20 unfiltered, with the reranker on and off. The reranker was the same
small open model, on the CPU, over the 64-candidate pool.

**Figure: Median search time, filtered queries.** The reranker added 3,478 ms at 2.4K memories and 3,564 ms at 10K, 92% and 86% of the search time.

| Item | Value (ms) |
|---|---|
| 2.4K memories, reranker off | 315 |
| 2.4K memories, reranker on | 3793 |
| 10K memories, reranker off | 560 |
| 10K memories, reranker on | 4124 |

Source: Esment Lab, October 2026
Two things stand out. First, **the reranker's cost barely changed with the
size of the store**: its work depends on the 64 candidates, not on how many
memories exist. What grew with the store was everything else, from 315 to 560
ms. Second, the cost was about seven times the 500 ms budget we had set for a
search.

This test has real limits. Only 5 and 21 distinct questions existed at the two
sizes, so the 40 filtered queries repeated texts. We did not rule out query caching
by measurement. The machine was swapping heavily,
which inflates the tails, most of all for unfiltered queries on the larger
set, where the noise swamped the reranker's share. Treat the absolute times as
upper bounds and the ratios as the finding.

## What we changed because of it

The ablation said keep the reranker. The latency test said it was too slow to
keep as it was. We made it cheaper rather than smaller, because shrinking the
candidate pool to 32 or 16 cut time but also cut end-to-end accuracy.

- **Same engine for assistants.** Along the way we found that the search tool assistants call skipped the vector stage and the reranker, so it was not the engine we had measured. We made it use the same path. Through that tool, accuracy on the same 150 questions went from 58.0% to 73.3%, matching the no-reranker result above.
- **A faster local reranker.** We sorted candidates by length into batches and stopped requests from queuing behind each other. On the evaluation corpus, accuracy stayed at 78.7% with no discordant answers, the top ten was identical on 98.7% of questions, and the median reranker time fell from 2,053 to 1,176 ms.
- **A time limit.** If the reranker fails or takes longer than 3 seconds, search returns the order from the fused pool instead of an error.
- **A cheaper cascade.** Once the reranker was faster, the pool-of-64 cascade became the bottleneck on that corpus, at 7.9 seconds median. We fixed that too, down to 0.27 seconds, with accuracy unchanged.
- **One thing we did not ship by default.** An 8-bit version of this English reranker was smaller but scored 75.3%, so we did not make it the default.

Those were measured on the evaluation corpus, not at the user-scale sizes
above, so don't subtract them from the 3.5 seconds. They are what lies behind
the faster, fully local search described in
[Esment 1.8.0](https://esment.notas.ai/newsroom/retrieval-v2-faster-multilingual-memory-search).

## What we still don't know

- Whether the reranker's gain is real at the size we measured. Five points with p = 0.15 is a lean, not a result.
- How the reranker behaves outside English, which we tested in [our multilingual memory search note](https://esment.notas.ai/blog/multilingual-memory-search). The benchmark and the reranker we tested are both English.
- What search time looks like for real people. Our user-scale runs used a few repeated questions on a machine under pressure, and we haven't published a fresh run at those sizes since the changes.
- Whether better retrieval is the main fix. When we read wrong answers, many had the evidence in view, as in [why AI memory answers fail](https://esment.notas.ai/blog/why-ai-memory-answers-fail).
- Whether benchmark questions look like yours. We wrote about that gap in [the benchmark is not the territory](https://esment.notas.ai/research/the-benchmark-is-not-the-territory).

That is the other half of why we publish the conditions. As of November 2026,
retrieval in Esment is a stack we have taken apart, not just one we have
assembled. If you are new to the vector stage that this stack starts from,
[what is an embedding](https://esment.notas.ai/learn/what-is-an-embedding) is the place to begin, and
[how Esment works](https://esment.notas.ai/learn/how-esment-works) shows where search sits among the
rest.

## Questions, answered

### What is an ablation in AI memory retrieval?

An ablation turns one part of a system off and measures what changes. For a retrieval cascade that means running the same questions with the graph stage off, the reranker off, or both, then comparing recall and speed against the full setup.

### Does a reranker improve recall?

In our test it barely moved recall, 92.7% against 91.3% on full recall. Its effect showed up later, in the answers a model wrote from the top ten results, where accuracy dropped about five points without it.

### Why is a reranker slow?

It reads the question together with every candidate, one pair at a time, instead of comparing precomputed vectors. With 64 candidates on a CPU, that was about 3.5 seconds in our user-scale test, however large the memory store was.

### Are these numbers what Esment users see?

No. They come from one development machine under memory pressure, on a corpus built from benchmark conversations. Only the differences between configurations carry meaning. The absolute times and scores are not production numbers.
