Run a local model
Use an open-source model on your Mac instead of a cloud provider, with Ollama or a GGUF file.
Esment and the LLM
Esment is a memory engine, so it doesn’t contain an LLM (a large language model, the kind of AI that writes text). Two features do need one: the Assistant (the in-app chat) and conversation extraction (the worker that turns an imported conversation into memories). In Settings → Intelligence, the Where Esment runs AI menu chooses where those run: On this Mac, OpenAI or Ollama.
Both local choices keep the work on your Mac: same engine, same memory, no calls to an AI provider. The embeddings that power search already run locally with FastEmbed.
Two ways to go local
Ollama (recommended for most people)
Ollama is a free app that downloads and runs models for you. Choose Ollama under Where Esment runs AI and Esment talks to it for chat and embeddings. It’s the shortest path and the easiest to maintain.
Direct GGUF with llama.cpp (the Esment-native way)
Esment can launch a llama-server subprocess (part of llama.cpp) for the extraction workers. You drop a GGUF model file into ~/.esment/models/, switch it on, and the engine starts its own local server on port 8181. The model is not inside the app, and the app tells you if it can’t find llama-server.
Install & connect Ollama
- 1Install Ollama from ollama.com/download and launch it.
- 2Pull a model that fits your Mac, see the next section. For example:
ollama pull qwen3:8b. Also pull the embedding model Esment asks Ollama for:ollama pull nomic-embed-text. - 3Open Esment → Settings → Intelligence.
- 4Set Where Esment runs AI to Ollama, check Ollama URL (default
http://127.0.0.1:11434) and click Apply. The engine restarts. - 5To choose which model the Assistant uses, click Advanced…, set Assistant provider to Ollama · local, type the model name you pulled and click Save assistant. The default name there is
qwen3.6:27b, which is large. - 6Ask the Assistant something. It should answer from the local model with no cloud call.
Which model to pull
A good starting point for a Mac with 16GB of RAM is a 7–9B model — noticeably more accurate at extracting facts from conversations than the 3B class, and still fast. See the full list in the list below.
Download & drop a model
The GGUF format is the file llama.cpp reads — one file, no installation. Every major open model publishes official GGUF quantizations on Hugging Face.
- 1Pick a model from the list below and download its GGUF (q4_k_m recommended).
- 2Create the models folder:
mkdir -p ~/.esment/models(Enterprise:~/.esment-enterprise/models) and copy the file there, e.g.cp qwen2.5-3b-instruct-q4_k_m.gguf ~/.esment/models/. Or click Open folder under Models folder in Settings → Intelligence (the Local model row under On this Mac). The app detects the file on its own. - 3Make sure llama-server is installed. The Local model row shows a green check with its path, or tells you to run
brew install llama.cpp. - 4Set Where Esment runs AI to On this Mac and click Apply, or switch on Run extraction on a local model. If several models are in the folder, click Use on one (the active one is marked active), or Choose… for a file elsewhere.
- 5Click Restart now when it asks. The conversation worker now extracts memories locally.
Those controls edit the file below for you. To point the Assistant at the same local server instead of Ollama, open Advanced… and use an OpenAI-compatible address such as http://127.0.0.1:8181/v1. We have not verified that path end to end.
Enable it in the config
[local_llm]
# Start a bundled llama-server subprocess for conversation extraction
# (no OpenAI, no Ollama). Model is auto-detected from ~/.esment/models/.
enabled = true
binary_path = "" # empty = auto-detect llama-server (bundle or PATH)
model_path = "" # empty = auto-detect the .gguf in ~/.esment/models/
port = 8181
ctx_size = 4096
threads = 0 # 0 = auto-detect~/.esment-enterprise instead of ~/.esment. Esment still needs the network to validate your license.Sizes for your Mac
“Mid-capable on a good computer” in 2026 means roughly a 7–32B model on 16–64GB Macs. As a rule of thumb (GGUF q4):
- 8GB RAM — 1–4B models. Usable, best for simple extraction.
- 16GB RAM — 7–9B comfortably; 12B (MoE/quantized) tight.
- 24–32GB RAM — 14–32B dense, or 30B-class MoE — the sweet spot for extraction quality.
- 64GB+ RAM — 70B-class and MoE models like gpt-oss-120b.
What’s current (2026)
The open-weight landscape moves fast. These are families worth trying, with official download pages:
Model names and versions below, and the rough memory sizes, are from our notes and have not been re-checked against each release. Treat them as a starting point.
- Qwen (Qwen3.5 / Qwen3.6) — strong generalist and coding, excellent GGUF support. Official: huggingface.co/Qwen.
- Gemma 4 — compact and efficient; the 12B fits 16GB Macs. Official: huggingface.co/google.
- GLM (GLM-5.1) — a family with small and large sizes. Official: huggingface.co/zai-org.
- gpt-oss (20B / 120B) — OpenAI’s open-weight series; the 120B needs a big machine (~66GB+). Official: huggingface.co/openai.
- The recommended default — Qwen2.5-3B (q4_k_m, ~2 GB). Neither Esment app carries a model inside the download; get it here and drop it in the models folder (
~/.esment/models, or~/.esment-enterprise/modelsfor Enterprise): huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF.
New releases land every week — check Hugging Face trending or the Ollama library for what shipped most recently, and grab the GGUF quantization from the model’s official repo.
Quantization, explained
A GGUF file is a quantized version of the model: the weights are stored at reduced precision to save memory and speed up inference. The name encodes it — q4 = 4 bits per weight, q3 = 3 bits, and k_m is a smart mixed-precision scheme.
- q4_k_m — the standard. Best quality/size balance; what most tools default to.
- q3_k_m — ~20% smaller, a small quality dip. Fine for extraction and summarization.
- q8_0 — near-lossless, roughly twice the size of q4. For big machines.
- q2_k — smallest, visibly worse. Avoid for production use.
For Esment’s workloads (turning conversations into structured memories, summarizing, classifying) q4_k_m is the right default; q3_k_m is a reasonable trade-off on tight machines.