Multilingual memory search: how we tested it and what we changed
Esment Lab note on multilingual memory search: how we built the test set, what a cross-language test showed, and what we changed. Limits included.
In short
- Our English-only search stack did little outside English. On a public multilingual set, an English-only reranker scored 45.1 nDCG@10 against 38.6 for no reranker; a multilingual one scored 71.4.
- With questions in Spanish, German or French over English memories, answer accuracy fell from 78.7% to 64.4% with the English-only reranker, and held at 72.7% with the multilingual one.
- The multilingual reranker lost 2.7 points in English (76.0% vs 78.7%), inside the noise of a 150-question test but not proven harmless.
- We did not measure the exact case in the title, a memory saved in Spanish and found by an English question. Our questions were translated by a model, with no native review.
A memory saved in Spanish should be found by a question in English, and the other way round. That is multilingual memory search, and for a long time ours worked well in one language and unevenly in the rest. This Lab note is about how we measured that, what the measurements said, and what we changed. Some numbers are final. Some questions are still open, and we say which.
All figures are from October 2026, from Esment Lab's development machine. They compare configurations to each other. They are not production numbers.
Why a Spanish memory can miss an English question§
Search by meaning works because an embedding places each text in a space where similar meanings sit close together. A question gets the same treatment, and the nearest memories come back. Across languages that only works if the model learned one space for all of them. Our earlier embedder was a small English one. In another language it still produced vectors, but they carried less of the meaning.
Our search is also a cascade, as we described in what each stage of the retrieval cascade is worth. Several methods find candidates, then a second model, the reranker, reads the question next to each candidate and sorts them. That post ended with an open item: both the benchmark and the reranker we tested were English. This is the follow-up.
How we tested multilingual memory search§
We needed two kinds of test. The first checks a model in isolation: given a question and 64 candidate passages, can it put the right ones on top? For that we used an open multilingual retrieval dataset with human relevance labels, covering nine languages: Spanish, French, German, Japanese, Chinese, Arabic, Russian, Hindi and English.
For each language we kept questions with at least 20 candidates and at least one correct passage among the first 64, then sampled 150 per language with a fixed seed. Candidates stayed in their original order, so "no reranker" is a real baseline. On that set, the order alone scored 44.4 nDCG@10 on average across the nine languages, against roughly 5 to 10 for a random order.
The second test is end to end. We took the 150 English questions from our usual memory benchmark and translated them into Spanish, German and French, keeping the memories in English. A reader model saw the retrieved memories plus the original English question, and a judge model scored the answer. That design isolates one thing: what happens to retrieval when the question is not in English.
How good were the translations?§
Not checked by a native speaker. A language model translated the questions and then translated them back, and we compared the round trip with the original. Similarity was 0.95 for Spanish and 0.90 for German on a sample of 10, and 0.81 to 0.85 for Japanese. The checks caught drift in a few phrases ("coffee mug" became "cup of coffee"; "quote" became "offer"). The same model did both directions, so shared mistakes could go unnoticed. We did not report a round-trip check for French.
What the English-only reranker did outside English§
We measured four languages (English, Spanish, German, French) with 150 questions each, and scored the order the reranker produced with nDCG@10, where 100 is a perfect top ten. The English-only reranker helped in English. In Spanish and German it scored below the untouched order, and in French it was a little above it. Differences there were within the margin of error.
A small multilingual reranker running on the CPU scored 71.4 on average, about 33 points above no reranker (95% interval +29.8 to +35.9).
Reranking quality across English, Spanish, German, French
| No reranker | 38.6 nDCG@10 |
|---|---|
| English-only reranker | 45.1 nDCG@10 |
| Multilingual reranker | 71.4 nDCG@10 |
A larger multilingual reranker scored about the same as the small one (72.3 on the same average) at two to four times the compute, so we did not pursue it. The small model in full precision scored 71.9, so the 8-bit build lost little. We also tested shortening each candidate and shrinking the pool: cutting candidate text from 256 to 128 tokens lost 3.9 points, and a pool of 32 instead of 64 lost 6.5.
What the cross-language test showed§
On the end-to-end test, the English-only reranker kept 78.7% of answers correct when the question was in English. With the question in Spanish, German or French over the same English memories, it fell to 64.4% on average. The multilingual reranker scored 76.0% in English and 72.7% across the three other languages.
Answers judged correct, 150 questions per language
| English-only, English question | 78.7% |
|---|---|
| English-only, es/de/fr question | 64.4% |
| Multilingual, English question | 76% |
| Multilingual, es/de/fr question | 72.7% |
| Question language | English-only reranker | Multilingual reranker |
|---|---|---|
| Spanish | 102/150 = 68.0% | 112/150 = 74.7% |
| German | 94/150 = 62.7% | 110/150 = 73.3% |
| French | 94/150 = 62.7% | 105/150 = 70.0% |
| Mean | 64.4% | 72.7% |
Averaged over the three languages the multilingual reranker was 8.2 points ahead (95% interval +4.0 to +12.4, paired test p = 0.0002). The English-only reranker's drop from its own English score was significant in each language. The multilingual one's drop was not, in any of the three.
What it cost in English§
It was not free. In English the multilingual reranker got 114 of 150 right (76.0%) against 118 (78.7%). We had set a bar of 77.7% for English, so it missed by three questions. A second pass of the judge on the same retrieved contexts gave 73.3% and 78.0%, and averaging both passes put the difference at −3.7 points (interval −8.3 to +1.0).
The judge is itself noisy. Re-judging the same contexts moved 13 questions for one setup and 12 for the other. A sample of 150 cannot resolve a gap of three or four points, so we call this a likely small cost, not a measured one.
On speed, the multilingual reranker was faster, not slower. On 12 real question pools (64 candidates, 256 tokens each) on an Apple Silicon machine, the median rerank time was 0.57 seconds against 0.84 for the English-only one, and the file was 119 MB against 144 MB. Through the full search path, the median rerank was 579 ms against 869 ms, with no search falling back to the fused order on our time limit.
Linear arithmetic on the cross-language results, using the 3.7-point English cost averaged over both judge passes, puts the break-even at around 31% of questions in a language other than English. That is a rough figure from one setup, not a forecast of anyone's usage.
The embedder changes that matter more for non-English memories§
The reranker only reorders what retrieval found. If the embedder puts a Spanish memory far from its question, nothing downstream can rescue it. So we ran the same kind of test on embedders: 100 questions per language, 64 candidates each, same sample for every model.
Embedding quality across English, Spanish, German, French
| Order of the pool | 39.4 nDCG@10 |
|---|---|
| Small English embedder | 45.1 nDCG@10 |
| Small multilingual embedder | 68.8 nDCG@10 |
The English embedder scored 41.5 in Spanish and 36.3 in German, below the pool's own order (47.5 and 40.4), and 39.5 in French, above its 31.4. The multilingual one scored 73.7, 67.1 and 64.9. Averaged, it was 23.7 points better (interval +20.6 to +26.6), and in English alone 6.2 points better (interval +1.6 to +10.4). The 8-bit build matched full precision, within 0.1.
Three things made it cheap enough to ship locally. On 1,000 real memories on our test Mac, the multilingual model on our own inference engine embedded 57.9 memories per second with a 0.82 GB peak, against 12.7 per second and 4.2 GB for the previous setup. The file is 118 MB. Re-embedding about 73,600 memories took roughly 30 minutes of compute in our last run, against 2 hours 7 minutes for the old way.
What shipped, and what we left alone§
We chose the small multilingual embedder as the local default. Switching embedders does not by itself re-embed old memories, and two spaces with the same number of dimensions can mix silently. So when the model changes, the Mac app now sets the old vectors aside and re-embeds every memory in the background, most important and most recent first; a memory only counts in vector search again once its new vector lands. The reranker also needs its scores rescaled before they are combined with a memory's importance, because its raw scores span a wider range than the old one's.
What users got, as of November 2026, is in the Esment 1.8.0 announcement: search across Spanish, English, German, French and Italian, including a memory saved in one language and asked about in another, on the Mac, offline (see why local-first matters). The part about saving in one language and asking in another rests on our Spanish, German and French questions over English memories and on the models being trained for it; we have not run the mirror case. This note is the evidence behind that direction, not a spec for each release.
Our last check of the all-local stack, multilingual embedder plus multilingual reranker, scored 70.7% on the English questions and 66.7% on the Spanish ones. That is a tie with our earlier local stack in English (72.0%, p = 0.80), but below the 78.7% of the hosted-embedding setup we had used as the English bar. No all-local stack we tried reached that bar.
What we still don't know§
- The mirror case. Spanish memories with English questions, and memories and questions both in another language, were not run end to end. The second case is covered only by the model-level tests above.
- Italian. It is in the product, and both multilingual models list it, but we did not measure it.
- Translation quality. One model translated, back-translated and checked its own work. A native reviewer could change the numbers for any language.
- Other machines. We measured an Apple Silicon Mac. The 8-bit builds were validated there, and we did not measure quality on Intel.
- A gap in English. Three to four points is below what 150 questions can settle, so we are still measuring it.
Those are the same limits we gave in the benchmark is not the territory: a test is a model of use, and ours changed language before it changed anything else. For another failure we traced by reading the wrong answers, see why AI memory answers fail.
Questions, answered§
Can an AI memory find a note saved in another language?
Only if the search compares meaning rather than words, and only if its models were trained on more than one language. An English-only model has no reason to place a Spanish sentence near its English equivalent. A multilingual one is built to.
What is cross-lingual retrieval?
It is searching in one language and finding text written in another. The question and the stored memory never need to share a word. They only need to land near each other in the same space of meaning, which an embedding model provides.
Does a reranker need to be multilingual too?
In our tests, yes. The English-only reranker we used before did not reliably beat leaving the candidate order alone on Spanish, German and French documents. It read the pairs but had little to go on, so the order it produced was not an improvement.
Are these numbers what I will see in Esment?
No. They are Lab numbers from a development machine, on public question sets and a corpus built from benchmark conversations. Compare the configurations with each other. Do not read the percentages as a promise about your own memories.
Pass it on →