Essays4 min read

Why AI assistants need a memory they can't lose

Every AI assistant starts each chat from zero. Why bigger context windows don't fix that, and why real assistant memory must be inspectable, local and reliable.

The Esment Teamrevised

In short

  • Every assistant starts a new chat from zero, so you pay a "forgetting tax" — re-explaining, re-correcting, re-deciding.
  • A bigger context window doesn't remember anything; it only lets you paste more. The window is a cache, not a memory.
  • A memory worth trusting is inspectable (you can read it), local (it lives with you) and deterministic (recall isn't left to the model's whim).
  • The payoff compounds — the longer an assistant remembers you, the more useful each conversation gets.

The most valuable thing a good assistant has isn't the model. Models are replaced every few months and get better each time. What makes an assistant yours is the context — everything you've told it about your work, your preferences and your decisions — and today that context evaporates the moment a conversation ends.

We spent a year building Esment around that observation. This is the argument for why AI assistants need a memory, and what kind of memory it has to be.

The forgetting tax§

Every assistant you use starts from zero on every new chat. The cost isn't theoretical — you pay it every day:

  • you re-explain your preferences, your project layout, your constraints;
  • the assistant repeats mistakes you already corrected;
  • decisions made in one conversation are invisible in the next;
  • and the assistant you trust with real work — the one that has known you longest — is exactly the one with no memory of you.

Call it the forgetting tax. It is small per conversation and enormous over a year, and it grows with how much you rely on the assistant.

Starting a chat

Without memory–Re-explain the project–Re-state preferences–Repeat old corrections–Re-decide settled thingsWith memory✓Picks up the project✓Knows how you work✓Remembers corrections✓Builds on decisionsvs

Without memory: Re-explain the project; Re-state preferences; Repeat old corrections; Re-decide settled things. With memory: Picks up the project; Knows how you work; Remembers corrections; Builds on decisions.

The forgetting tax: what every new conversation costs when nothing carries over.

Some assistants now save a few facts between chats — ChatGPT's saved memories, Claude's memory and chat search. They help, and we cover how to get the most out of them in our guide to AI with memory. But each one is a memory of one assistant, kept on that company's servers, and recalled only when the model thinks to look.

Why a bigger context window doesn't fix it§

The industry's answer has mostly been longer context windows. A larger window is useful, but it doesn't remember anything: it lets you re-paste more. When the chat ends, it is cleared. While the chat goes on, it fills up, and older material gets squeezed out or compressed.

The window is a cache, not a memory. It holds what the model is looking at right now. Memory is what survives between conversations — chosen, stored and brought back on purpose. If you want the mechanics, the context window chapter explains what fits in one and why it runs out. Deciding what goes into that window is the job of context engineering.

One long conversation

context windowInstructionsPasted backgroundThis chatFree

Instructions (10), Pasted background (34), This chat (46), Free (10)

Without memory, background has to be pasted into every conversation — and it competes for space with the conversation itself.

What a memory has to be§

Once you accept that an assistant should keep what it learns, the design questions get sharp. These are our answers.

Inspectable§

If a system remembers things about you, you should be able to open it and see exactly what it knows. Esment stores every memory in a database on your machine with a full history: every change is a commit you can read and undo. "What do you know about me?" isn't the model's guess — it's a query. You can see the same idea in practice in find, edit and forget.

Local§

Your assistant's memory is a record of your work, your relationships and your decisions. Sending it to a service you don't control turns a tool into a liability. The Esment store runs on your device; the cloud is an opt-in mirror, never the default. We make the full case in local-first AI memory.

Deterministic§

The deepest failure of assistant memory isn't storage — it's recall. A model that only searches its memory when it thinks to search will, on average, not think to. So Esment brings a short session brief into every conversation before the model answers — the people, projects and decisions that matter right now — retrieved in tens of milliseconds. Recall stops being the model's decision and becomes a guarantee.

Memory has to change, too§

A memory that only grows becomes noise. People change jobs, projects end, preferences flip. A useful memory notices when a new fact replaces an old one, keeps the history, and lets the stale version fade from everyday recall. That's the hard engineering problem underneath all of this, and we wrote about how we approached it in building a temporal knowledge graph and, more formally, in a memory that cannot forget.

The compounding effect§

Memory changes the economics of using an assistant. The first week, the difference is small. By the third month, the assistant answers questions you haven't asked yet — because it knows the project, the constraints, the history. Alex, a product designer we use in our examples, stops starting chats with three paragraphs of background; the assistant already knows the design system, the deadline and who signs off.

What an assistant knows about you

Day 1Your name, your roleWeek 1How you like answersMonth 1Projects and peopleMonth 3Why decisions were made

Day 1: Your name, your role; Week 1: How you like answers; Month 1: Projects and people; Month 3: Why decisions were made

With memory, each conversation starts from everything before it — so the value compounds.

It stops being a chatbot and becomes a colleague with an excellent memory: annoying when it forgets, indispensable when it remembers.

The bet§

That is the bet Esment is making: that the winning assistants of the next decade won't be the ones with the biggest context windows, but the ones with the most reliable memory — and that the memory should belong to you, not to any one of them.

Questions, answered§

Don't AI assistants already have memory?

Some do, in a limited way. ChatGPT and Claude can save facts and search past chats, but each memory belongs to one assistant, lives on that company's servers and is recalled only when the model decides to look. Switch assistants and you start over.

Isn't a long context window the same as memory?

No. A context window is the text a model can read in one conversation. It is cleared when the chat ends and it fills up as the chat grows. Memory is what survives between conversations, and it has to be chosen, stored and brought back on purpose.

What should an AI memory store?

Durable things that make the next conversation better — your preferences, the projects you work on, the people involved, decisions and their reasons. Not every sentence you typed. Good memory is selective, dated and correctable.

Why does it matter where the memory lives?

Because it is a record of your work, relationships and decisions. A memory on your own machine can be inspected, backed up and deleted by you, and it can be shared with every assistant you use instead of being locked into one.

Pass it on →

Keep reading

all posts →
Essays2 min

Local-first AI memory is a feature, not a constraint

Why Esment keeps your AI memory on your own device — the privacy asymmetry, the architecture that keeps it fast, and how sync works without giving up ownership.

#local-first #privacy
Esment Lab3 min

What we learned building a temporal knowledge graph for AI memory

Why a flat list of facts fails as AI memory, how a temporal knowledge graph handles contradictions, and the four-stage retrieval cascade behind Esment.

#knowledgegraph #retrieval
Agents & context7 min

Context engineering: what it is, where memory fits, and a practical checklist

Context engineering explained: where the term came from, the parts of a context, how it fails, where durable memory fits, and a checklist for ten minutes.

#contextengineering #AGENTS.md

a note for you

New posts, by email.

Fixes, guides and Lab notes on AI memory — about two a month. No product news, unsubscribe in one click.

Or follow the RSS feed