3 sheets tied with this thread.
Esment Lab note on multilingual memory search: how we built the test set, what a cross-language test showed, and what we changed. Limits included.
A memory saved in Spanish should be found by a question in English, and the other way round. That is multilingual memory search, and for a long time ours worked well in one language and unevenly in the rest. This Lab note is about how we measured that, what the measurements said, and what we changed. Some numbers are final. Some questions are still open, and we say which.
We switched off the graph, the reranker and both in Esment's retrieval cascade: what each did to recall, what the reranker cost in time, and what we changed.
A memory search is not one search. Ours is a cascade: several ways of finding candidates, then a second model that puts them in order. This is an ablation of that retrieval cascade: we turned the graph off, turned the reranker off, then turned off both, and measured what each stage was worth in recall and in time. It is a Lab note, so it includes what we got wrong and what we still don't know.
We read the wrong answers from Esment's memory search. Most had the evidence in front of the model, which points at how memories get written, not found.
When a memory system gives a wrong answer, there are two places to look. Either the right memory never came back (a retrieval failure), or it came back and the model still got the answer wrong (a reading failure). The fix for each is different, so we sorted our own wrong answers into those two piles. This is why AI memory answers fail, as far as Esment's October search showed it, and what it changed about how we think memories should be written. It is a Lab note: we say what we measured, what we did about it, and what we don't know.
We use strictly-necessary cookies to run Esment, and — only with your consent — analytics cookies to improve it. Your memories stay on your device and are never part of this. See our Cookie Policy.