Essays2 min read

Local-first AI memory is a feature, not a constraint

Why Esment keeps your AI memory on your own device — the privacy asymmetry, the architecture that keeps it fast, and how sync works without giving up ownership.

The Esment Teamrevised

In short

  • A memory store is the one dataset where what you want (an assistant that knows you) and what a vendor wants (to know you) pull apart.
  • Keeping memory on your machine removes that tension — nothing to misuse, leak or hand over.
  • The engine is one native binary with a local database, a knowledge graph, a four-stage retrieval cascade and an MCP server, answering in milliseconds.
  • Sync is optional and additive: the cloud can mirror your memory, but it never owns it.

There's a moment in every memory-product pitch where someone asks: "but why not just put it in the cloud?" It's a fair question. The cloud is easier to build, easier to sync and easier to sell. We chose local-first AI memory anyway, and the reasons aren't philosophical — they're practical.

The privacy asymmetry§

A memory store is the one dataset where the user's interests and the company's incentives are structurally misaligned. You want the assistant to know you; the company wants to know you. Every cloud memory product has to be judged under that tension, however good its intentions.

A local store dissolves it. When the data never leaves the machine, there's nothing to misuse, nothing to subpoena and nothing to leak. That's not a promise in a privacy policy — it's a property of where the file is.

Where your memory lives

Cloud-first–Vendor holds the data–Trust the policy–Network on every recall–Locked to one serviceLocal-first✓You hold the data✓Check it yourself✓Recall on the device✓Shared by every assistantvs

Cloud-first: Vendor holds the data; Trust the policy; Network on every recall; Locked to one service. Local-first: You hold the data; Check it yourself; Recall on the device; Shared by every assistant.

The same feature — an assistant that remembers you — with the data in two very different places.

This matters more for memory than for most data, because of what memory is: a running record of your work, your relationships and your decisions. If you're weighing the options, what "local" really means for an MCP memory server is a good checklist.

The architecture§

The Esment engine is a single native binary. On your machine it runs a local database with:

  • a temporal knowledge graph — entities, relations and the memories that ground them (how we built it);
  • a four-stage retrieval cascade — full-text, vectors, graph expansion and a diversity-aware rerank (what each stage is worth);
  • a commit history — every change is an auditable commit you can inspect and undo;
  • an MCP server — any assistant that speaks the Model Context Protocol can connect to it.

On your machine

Your assistants · Claude,ChatGPT, CursorMCP serverRetrieval cascadeKnowledge graph + historyLocal database file

Your assistants · Claude, ChatGPT, Cursor, then MCP server, then Retrieval cascade, then Knowledge graph + history, then Local database file

Everything from storage to the assistant connection runs locally.

All of it answers in milliseconds. The search index is built locally; the calls to an AI model are the only thing that touch a network, and only to the provider you choose. For the whole picture, see how Esment works.

The sync question§

Local-first doesn't mean isolated. Esment syncs both ways with its hosted cloud when you ask it to: additive in both directions, safe against conflicts, with search indexes rebuilt on each side. The cloud becomes a mirror of your memory, never the owner of it. Esment Cloud covers what syncs and how to turn it off.

Why this wins§

  • Speed. No round trip to a server for every recall; the cascade runs on the device in tens of milliseconds.
  • Portability. The store is a file. Back it up, copy it, move it. It's yours in a way a row in someone else's database never is.
  • Honesty. "Your memory stays on your machine" is a claim you can check yourself. The enterprise edition goes further: it ships fully local, with no outside model providers and no network at all — a memory engine that runs on an air-gapped machine.

Local-first is often described as a constraint we accepted. It isn't. It's the feature that makes the whole product trustworthy enough to use — and the reason one memory can be shared by every assistant you use without belonging to any of them.

Questions, answered§

What does local-first AI memory mean?

It means the memory your assistants use is stored and searched on your own device first. The cloud, if you use it at all, is a copy you choose to keep — not the place your data lives by default.

Is local AI memory slower than cloud memory?

Usually the opposite. Recall doesn't need a round trip to a server, so the retrieval cascade runs on the device in tens of milliseconds. The only network calls are to the AI model provider you configure.

Can local-first memory still sync between devices?

Yes. Esment can sync with its hosted cloud when you turn it on. Sync is additive in both directions and safe against conflicts, and the cloud copy is a mirror of your local store.

Can I use a local AI memory with ChatGPT and Claude?

Yes. Assistants connect to Esment through MCP and connectors, so Claude, ChatGPT, Cursor and others read the same memory while it stays on your machine.

Pass it on →

Keep reading

all posts →
Esment Lab3 min

What we learned building a temporal knowledge graph for AI memory

Why a flat list of facts fails as AI memory, how a temporal knowledge graph handles contradictions, and the four-stage retrieval cascade behind Esment.

#knowledgegraph #retrieval
Esment Lab6 min

What each stage of our memory retrieval cascade is worth

We switched off the graph, the reranker and both in Esment's retrieval cascade: what each did to recall, what the reranker cost in time, and what we changed.

#retrieval #evaluation
Essays4 min

Why AI assistants need a memory they can't lose

Every AI assistant starts each chat from zero. Why bigger context windows don't fix that, and why real assistant memory must be inspectable, local and reliable.

#memory #essays

a note for you

New posts, by email.

Fixes, guides and Lab notes on AI memory — about two a month. No product news, unsubscribe in one click.

Or follow the RSS feed