Esment Lab3 min read

What we learned building a temporal knowledge graph for AI memory

Why a flat list of facts fails as AI memory, how a temporal knowledge graph handles contradictions, and the four-stage retrieval cascade behind Esment.

The Esment Teamrevised

In short

  • A flat list of facts can't answer "how does this relate to that?" and can't handle contradiction — so it fails as assistant memory.
  • In a graph, memories are grounded in entities (people, projects, tools) and linked by typed relations, so retrieval can expand outward.
  • Every memory has a lifecycle; when a new fact contradicts an old one, the old one is archived, not deleted, so the past stays queryable.
  • Retrieval is a cascade — keywords, meaning, graph expansion, then a diversity-aware rerank — because no single technique is enough.

The first version of Esment treated memory as a flat list: facts in, facts out, ranked by recency and importance. It worked, and it was wrong. A flat list can't answer the questions that matter — how does this relate to that? — and it can't handle contradiction. This is what we learned replacing it with a temporal knowledge graph, and why the retrieval on top of it is a cascade rather than a single search.

Why a flat list of facts fails§

Facts aren't independent. "Alex prefers dark mode" and "Alex works on the Atlas redesign" are both true, but they're different kinds of truth: one is a preference attached to a person, the other an activity attached to a project. In a list, they're two rows. Ask about the redesign and the list can find rows that mention it — but not the person, the tools or the decisions around it unless they happen to share the same words.

Contradiction is worse. When Alex moves from the Atlas redesign to a new project, a list now holds two facts that can't both be current, and nothing in the data says which one wins.

Memories grounded in entities§

In a graph, memories become nodes grounded in entities, and relations become typed edges — WORKS_ON, USES, DEPENDS_ON, CREATES. Retrieval can then do something a keyword search can't: it expands. Ask about a project and the graph walks out to the people, tools and decisions connected to it.

From a question to its neighbours

AtlasredesignwhoAlexusesWORKS_ONFigmadecidedShip byMarch

Atlas redesign → Alex → Figma → Ship by March

Graph expansion: one hit on the project brings back the person, the tool and the decision linked to it.

The point isn't the data structure for its own sake. It's that "what's going on with the redesign?" deserves an answer that includes who, with what and what we decided, even when the question names none of them.

The lifecycle problem§

A memory that can't forget becomes noise. Every store needs a lifecycle, and ours has five states:

A memory's life

GeneratedActivatedMergedArchivedExpiredlifecycle

Generated → Activated → Merged → Archived → Expired → back to the start

Merging is the hard step: deciding which of two conflicting facts survives.

The subtle part is merging: when new information contradicts old information, which one survives? Our reconciler resolves conflicts in time — a newer fact supersedes an older one — but the old fact is never deleted. It's archived, and the commit history keeps exactly what was known at every point.

That turns the store into a time machine. What did we believe about this project in March? is a query, not a guess. It's also why supersession, not decay, is the default: a fact doesn't become less true because it's old, only because something newer replaced it. We worked through that distinction carefully in a memory that cannot forget.

We ended up with a four-stage pipeline because no single technique is enough:

  1. Full-text (BM25) — exact words and names, in milliseconds.
  2. Vectors — meaning, for things worded differently than you stored them.
  3. Graph expansion — walking the relations out from the best hits; the stage that makes the graph pay for itself.
  4. Diversity-aware reranking (MMR) — the final order, so the context the assistant gets isn't five near-copies of the same fact.

Each stage covers a blind spot of the one before. Keyword search misses paraphrases; vectors miss exact names and rare terms; neither knows that a person is connected to a project; and without a final rerank the top results tend to repeat each other. If embeddings are new to you, what is an embedding explains the vector step in plain terms.

The retrieval cascade

Full-text · exact wordsVectors · similar meaningGraph · connected factsRerank · diverse top results

Full-text · exact words, then Vectors · similar meaning, then Graph · connected facts, then Rerank · diverse top results

Four fast stages, each covering what the previous one misses.

Each stage is fast on its own; the cascade is what makes the whole thing feel like a memory rather than a search engine. Since this post was first written, the cascade got faster and multilingual — see retrieval v2.

The lesson§

Building a memory engine is 20% storage and 80% judgment: what to keep, what to merge, what to let die and what to surface. The graph gave us the structure to make those judgments well. The lifecycle gave us the courage to make them at all.

If you want the case for why any of this matters, start with why AI assistants need a memory they can't lose; for where the graph lives, read local-first AI memory. To see what each retrieval stage adds, read what each stage of our retrieval cascade is worth, and for failures that start at write time, why AI memory answers fail.

Questions, answered§

What is a temporal knowledge graph?

A knowledge graph stores facts as entities (people, projects, tools) connected by typed relations. A temporal one also records when each fact was true, so a newer fact can supersede an older one without erasing it, and you can ask what was believed at any point in time.

Why use a knowledge graph for AI memory instead of plain search?

Because facts aren't independent. A graph lets retrieval walk from a project to the people, tools and decisions connected to it, which plain keyword or similarity search can't do. It also gives contradictions a place to be resolved.

How does Esment handle contradictory memories?

A reconciler compares new information with what's stored. When a newer fact contradicts an older one, the newer one becomes current and the older one is archived with its history, so recall stays clean and nothing is silently lost.

Does a knowledge graph make retrieval slower?

Not in practice. Each stage of the cascade is fast on its own and runs on the device; the graph step only expands from a handful of candidates the earlier stages already found.

Pass it on →

Keep reading

all posts →
Esment Lab6 min

What each stage of our memory retrieval cascade is worth

We switched off the graph, the reranker and both in Esment's retrieval cascade: what each did to recall, what the reranker cost in time, and what we changed.

#retrieval #evaluation
Esment Lab8 min

Why AI memory answers fail: retrieval or reading?

We read the wrong answers from Esment's memory search. Most had the evidence in front of the model, which points at how memories get written, not found.

#memory #retrieval
Essays2 min

Local-first AI memory is a feature, not a constraint

Why Esment keeps your AI memory on your own device — the privacy asymmetry, the architecture that keeps it fast, and how sync works without giving up ownership.

#local-first #privacy

a note for you

New posts, by email.

Fixes, guides and Lab notes on AI memory — about two a month. No product news, unsubscribe in one click.

Or follow the RSS feed