---
title: Multilingual memory search: how we tested it and what we changed
description: Esment Lab note on multilingual memory search: how we built the test set, what a cross-language test showed, and what we changed. Limits included.
date: 2026-11-18
updated: 2026-11-18
author: The Esment Team
category: Esment Lab
tags: multilingual, retrieval, evaluation, memory
canonical: https://esment.notas.ai/blog/multilingual-memory-search
---

# Multilingual memory search: how we tested it and what we changed

*The Esment Team · 2026-11-18 · [HTML version](https://esment.notas.ai/blog/multilingual-memory-search)*

## In short

- Our English-only search stack did little outside English. On a public multilingual set, an English-only reranker scored 45.1 nDCG@10 against 38.6 for no reranker; a multilingual one scored 71.4.
- With questions in Spanish, German or French over English memories, answer accuracy fell from 78.7% to 64.4% with the English-only reranker, and held at 72.7% with the multilingual one.
- The multilingual reranker lost 2.7 points in English (76.0% vs 78.7%), inside the noise of a 150-question test but not proven harmless.
- We did not measure the exact case in the title, a memory saved in Spanish and found by an English question. Our questions were translated by a model, with no native review.

A memory saved in Spanish should be found by a question in English, and the
other way round. That is **multilingual memory search**, and for a long time
ours worked well in one language and unevenly in the rest. This Lab note is
about how we measured that, what the measurements said, and what we changed.
Some numbers are final. Some questions are still open, and we say which.

All figures are from October 2026, from Esment Lab's development machine. They
compare configurations to each other. They are not production numbers.

## Why a Spanish memory can miss an English question

Search by meaning works because an [embedding](https://esment.notas.ai/learn/what-is-an-embedding)
places each text in a space where similar meanings sit close together. A
question gets the same treatment, and the nearest memories come back. Across
languages that only works if the model learned one space for all of them.
Our earlier embedder was a small English one. In another language it still
produced vectors, but they carried less of the meaning.

Our search is also a cascade, as we described in
[what each stage of the retrieval cascade is worth](https://esment.notas.ai/blog/memory-retrieval-cascade-ablation).
Several methods find candidates, then a second model, the reranker, reads the
question next to each candidate and sorts them. That post ended with an open
item: both the benchmark and the reranker we tested were English. This is the
follow-up.

## How we tested multilingual memory search

We needed two kinds of test. The first checks a model in isolation: given a
question and 64 candidate passages, can it put the right ones on top? For that
we used an open multilingual retrieval dataset with human relevance labels,
covering nine languages: Spanish, French, German, Japanese, Chinese, Arabic,
Russian, Hindi and English.

For each language we kept questions with at least 20 candidates and at least
one correct passage among the first 64, then sampled 150 per language with a
fixed seed. Candidates stayed in their original order, so "no reranker" is a
real baseline. On that set, the order alone scored 44.4 nDCG@10 on average
across the nine languages, against roughly 5 to 10 for a random order.

The second test is end to end. We took the 150 English questions from our
usual memory benchmark and translated them into Spanish, German and French,
keeping the memories in English. A reader model saw the retrieved memories plus
the **original English question**, and a judge model scored the answer. That
design isolates one thing: what happens to retrieval when the question is not
in English.

> [!NOTE]
> This is not the case in this note's opening line. Our memories stayed in
> English and the questions changed language. A memory saved in Spanish and
> found by an English question is the mirror image, and we have not measured it
> end to end.

### How good were the translations?

Not checked by a native speaker. A language model translated the questions and
then translated them back, and we compared the round trip with the original.
Similarity was 0.95 for Spanish and 0.90 for German on a sample of 10, and
0.81 to 0.85 for Japanese. The checks caught drift in a few phrases ("coffee
mug" became "cup of coffee"; "quote" became "offer"). The same model did both
directions, so shared mistakes could go unnoticed. We did not report a
round-trip check for French.

## What the English-only reranker did outside English

We measured four languages (English, Spanish, German, French) with 150
questions each, and scored the order the reranker produced with nDCG@10, where
100 is a perfect top ten. The English-only reranker helped in English. In
Spanish and German it scored below the untouched order, and in French it was
a little above it. Differences there were within the margin of error.

A small multilingual reranker running on the CPU scored 71.4 on average, about 33
points above no reranker (95% interval +29.8 to +35.9).

**Figure: Reranking quality across English, Spanish, German, French.** Macro average over four languages, 150 questions each, 64 candidates per question. The multilingual model is a small 8-bit one.

| Item | Value (nDCG@10) |
|---|---|
| No reranker | 38.6 |
| English-only reranker | 45.1 |
| Multilingual reranker | 71.4 |

Source: Esment Lab, October 2026
A larger multilingual reranker scored about the same as the small one (72.3 on
the same average) at two to four times the compute, so we did not pursue it.
The small model in full precision scored 71.9, so the 8-bit build lost little. We also tested shortening each candidate and shrinking
the pool: cutting candidate text from 256 to 128 tokens lost 3.9 points, and
a pool of 32 instead of 64 lost 6.5.

## What the cross-language test showed

On the end-to-end test, the English-only reranker kept 78.7% of answers
correct when the question was in English. With the question in Spanish, German
or French over the same English memories, it fell to 64.4% on average. The
multilingual reranker scored 76.0% in English and 72.7% across the three
other languages.

**Figure: Answers judged correct, 150 questions per language.** Memories stay in English; only the question changes language. Es/de/fr is the mean of three languages.

| Item | Value (%) |
|---|---|
| English-only, English question | 78.7 |
| English-only, es/de/fr question | 64.4 |
| Multilingual, English question | 76 |
| Multilingual, es/de/fr question | 72.7 |

Source: Esment Lab, October 2026
| Question language | English-only reranker | Multilingual reranker |
|---|---|---|
| Spanish | 102/150 = 68.0% | 112/150 = 74.7% |
| German | 94/150 = 62.7% | 110/150 = 73.3% |
| French | 94/150 = 62.7% | 105/150 = 70.0% |
| Mean | 64.4% | 72.7% |

Averaged over the three languages the multilingual reranker was 8.2 points
ahead (95% interval +4.0 to +12.4, paired test p = 0.0002). The English-only
reranker's drop from its own English score was significant in each language.
The multilingual one's drop was not, in any of the three.

## What it cost in English

It was not free. In English the multilingual reranker got 114 of 150 right
(76.0%) against 118 (78.7%). We had set a bar of 77.7% for English, so it
missed by three questions. A second pass of the judge on the same retrieved
contexts gave 73.3% and 78.0%, and averaging both passes put the difference at
−3.7 points (interval −8.3 to +1.0).

The judge is itself noisy. Re-judging the same contexts moved 13 questions for
one setup and 12 for the other. A sample of 150 cannot resolve a gap of three
or four points, so we call this a likely small cost, not a measured one.

On speed, the multilingual reranker was faster, not slower. On 12 real
question pools (64 candidates, 256 tokens each) on an Apple Silicon machine, the
median rerank time was 0.57 seconds against 0.84 for the English-only one, and
the file was 119 MB against 144 MB. Through the full search path, the median
rerank was 579 ms against 869 ms, with no search falling back to the
fused order on our time limit.

Linear arithmetic on the cross-language results, using the 3.7-point English
cost averaged over both judge passes, puts the break-even at around
31% of questions in a language other than English. That is a rough figure from
one setup, not a forecast of anyone's usage.

## The embedder changes that matter more for non-English memories

The reranker only reorders what retrieval found. If the embedder puts a Spanish
memory far from its question, nothing downstream can rescue it. So we ran the
same kind of test on embedders: 100 questions per language, 64 candidates each,
same sample for every model.

**Figure: Embedding quality across English, Spanish, German, French.** Macro average, 100 questions per language. The multilingual model is an 8-bit build; the same model in full precision scored 68.7.

| Item | Value (nDCG@10) |
|---|---|
| Order of the pool | 39.4 |
| Small English embedder | 45.1 |
| Small multilingual embedder | 68.8 |

Source: Esment Lab, October 2026
The English embedder scored 41.5 in Spanish and 36.3 in German, below the
pool's own order (47.5 and 40.4), and 39.5 in French, above its 31.4. The
multilingual one scored 73.7, 67.1 and 64.9. Averaged, it was 23.7 points
better (interval +20.6 to +26.6), and in English alone 6.2 points better
(interval +1.6 to +10.4). The 8-bit build matched full precision, within 0.1.

Three things made it cheap enough to ship locally. On 1,000 real memories on
our test Mac, the multilingual model on our own inference engine embedded 57.9
memories per second with a 0.82 GB peak, against 12.7 per second and 4.2 GB for
the previous setup. The file is 118 MB. Re-embedding about 73,600 memories
took roughly 30 minutes of compute in our last run, against 2 hours 7 minutes
for the old way.

## What shipped, and what we left alone

We chose the small multilingual embedder as the local default. Switching
embedders does not by itself re-embed old memories, and two spaces with the
same number of dimensions can mix silently. So when the model changes, the Mac
app now sets the old vectors aside and re-embeds every memory in the background,
most important and most recent first; a memory only counts in vector search
again once its new vector lands. The reranker
also needs its scores rescaled before they are combined with a memory's
importance, because its raw scores span a wider range than the old one's.

What users got, as of November 2026, is in the
[Esment 1.8.0 announcement](https://esment.notas.ai/newsroom/retrieval-v2-faster-multilingual-memory-search):
search across Spanish, English, German, French and Italian, including a memory
saved in one language and asked about in another, on the Mac, offline (see [why local-first matters](https://esment.notas.ai/blog/local-first-ai-memory)). The
part about saving in one language and asking in another rests on our
Spanish, German and French questions over English memories and on the models
being trained for it; we have not run the mirror case. This note is the
evidence behind that direction, not a spec for each release.

Our last check of the all-local stack, multilingual embedder plus multilingual
reranker, scored 70.7% on the English questions and 66.7% on the Spanish ones.
That is a tie with our earlier local stack in English (72.0%, p = 0.80), but
below the 78.7% of the hosted-embedding setup we had used as the English bar.
No all-local stack we tried reached that bar.

## What we still don't know

- **The mirror case.** Spanish memories with English questions, and memories
  and questions both in another language, were not run end to end. The
  second case is covered only by the model-level tests above.
- **Italian.** It is in the product, and both multilingual models list it, but we
  did not measure it.
- **Translation quality.** One model translated, back-translated and checked its
  own work. A native reviewer could change the numbers for any language.
- **Other machines.** We measured an Apple Silicon Mac. The 8-bit builds were
  validated there, and we did not measure quality on Intel.
- **A gap in English.** Three to four points is below what 150 questions can
  settle, so we are still measuring it.

Those are the same limits we gave in
[the benchmark is not the territory](https://esment.notas.ai/research/the-benchmark-is-not-the-territory):
a test is a model of use, and ours changed language before it changed anything
else. For another failure we traced by reading the wrong answers, see [why AI memory answers fail](https://esment.notas.ai/blog/why-ai-memory-answers-fail).

## Questions, answered

### Can an AI memory find a note saved in another language?

Only if the search compares meaning rather than words, and only if its models were trained on more than one language. An English-only model has no reason to place a Spanish sentence near its English equivalent. A multilingual one is built to.

### What is cross-lingual retrieval?

It is searching in one language and finding text written in another. The question and the stored memory never need to share a word. They only need to land near each other in the same space of meaning, which an embedding model provides.

### Does a reranker need to be multilingual too?

In our tests, yes. The English-only reranker we used before did not reliably beat leaving the candidate order alone on Spanish, German and French documents. It read the pairs but had little to go on, so the order it produced was not an improvement.

### Are these numbers what I will see in Esment?

No. They are Lab numbers from a development machine, on public question sets and a corpus built from benchmark conversations. Compare the configurations with each other. Do not read the percentages as a promise about your own memories.
