Esment logo
Esment
Chapter I

Run Esment with a local model

Esment is a memory engine — it doesn’t contain an LLM. But two features do need one: the native assistant (the chat that talks back to you) and conversation extraction (the worker that turns a pasted conversation into memories). By default the standard app sends those calls to a cloud provider — OpenAI, Anthropic, Mistral, Gemini — or to Ollama if you point it there.

You don’t have to send anything to the cloud. Esment can run its entire LLM workload on your Mac with an open-source model you download yourself — same engine, same memory, zero network calls.

Two ways to go local

Ollama (recommended for most people)

Ollama is a free app that downloads and runs models for you. Esment’s Settings → Assistant LLM has a built-in Ollama provider — pick it, pick a model, done. It’s the shortest path and the easiest to maintain.

Direct GGUF with llama.cpp (the Esment-native way)

Esment ships with support for a bundled llama-server subprocess. You drop a GGUF model file into ~/.esment/models/, flip one config flag, and the engine launches its own local server on port 8181. No extra apps installed. This is the same mechanism the Enterprise build uses — just with the model already included.

Install & connect Ollama

  1. 1Install Ollama from ollama.com/download and launch it.
  2. 2Pull a model that fits your Mac — see the table in the next section. For example: ollama pull qwen3:8b.
  3. 3Open Esment → Settings → Assistant LLM.
  4. 4Set Provider to Ollama, pick the model, save, and restart the local server when prompted.
  5. 5Ask the assistant something — it should answer from the local model with no cloud call.

Which model to pull

A good starting point for a Mac with 16GB of RAM is a 7–9B model — noticeably more accurate at extracting facts from conversations than the 3B class, and still fast. See the full list in Chapter IV.

Download & drop a model

The GGUF format is the file llama.cpp reads — one file, no installation. Every major open model publishes official GGUF quantizations on Hugging Face.

  1. 1Pick a model from the list in Chapter IV and download its GGUF (q4_k_m recommended).
  2. 2Create the models folder: mkdir -p ~/.esment/models and copy the file there, e.g. cp qwen2.5-3b-instruct-q4_k_m.gguf ~/.esment/models/.
  3. 3Edit ~/.esment/config.toml — enable the local LLM as shown below.
  4. 4Restart the Esment server from the app. The conversation worker now extracts memories locally.

The assistant chat itself is configured separately in Settings → Assistant LLM; point it at http://127.0.0.1:8181/v1 with model local (or the model name) to use the same local server.

Enable it in the config

[local_llm]
# Start a bundled llama-server subprocess for conversation extraction
# (no OpenAI, no Ollama). Model is auto-detected from ~/.esment/models/.
enabled = true
binary_path = ""   # empty = auto-detect llama-server (bundle or PATH)
model_path = ""    # empty = auto-detect the .gguf in ~/.esment/models/
port = 8181
ctx_size = 4096
Esment Note

Everything stays on this Mac: embeddings already run locally with FastEmbed, and with a local model the LLM calls do too — the only network Esment needs is for license validation.

Sizes for your Mac

“Mid-capable on a good computer” in 2026 means roughly a 7–32B model on 16–64GB Macs. As a rule of thumb (GGUF q4):

  • 8GB RAM — 1–4B models. Usable, best for simple extraction.
  • 16GB RAM — 7–9B comfortably; 12B (MoE/quantized) tight.
  • 24–32GB RAM — 14–32B dense, or 30B-class MoE — the sweet spot for extraction quality.
  • 64GB+ RAM — 70B-class and MoE models like gpt-oss-120b.

What’s current (2026)

The open-weight landscape moves fast. As of mid-2026 the families worth trying, with official download pages:

New releases land every week — check Hugging Face trending or the Ollama library for what shipped most recently, and grab the GGUF quantization from the model’s official repo.

Quantization, explained

A GGUF file is a quantized version of the model: the weights are stored at reduced precision to save memory and speed up inference. The name encodes it — q4 = 4 bits per weight, q3 = 3 bits, and k_m is a smart mixed-precision scheme.

  • q4_k_m — the standard. Best quality/size balance; what most tools default to.
  • q3_k_m — ~20% smaller, a small quality dip. Fine for extraction and summarization.
  • q8_0 — near-lossless, roughly twice the size of q4. For big machines.
  • q2_k — smallest, visibly worse. Avoid for production use.

For Esment’s workloads (turning conversations into structured memories, summarizing, classifying) q4_k_m is the right default; q3_k_m is a reasonable trade-off on tight machines.

Run a local model