Enterprise models

Esment uses two kinds of model. Search models (an embedder and a reranker) power retrieval and run inside the engine. A language model (an LLM) does the writing-heavy work and runs in llama-server (llama.cpp) on port 8181.

The Enterprise download does not include the language model. The app’s Settings → Intelligence panel reports No GGUF models found until you add one. Download a GGUF file (recommended: Qwen2.5-3B-Instruct, q4_k_m, about 2 GB) and put it in the models folder, ~/.esment-enterprise/models. Click Open folder in the panel to find it; the app detects the file and you click Use to activate it.

The language model powers the LLM-dependent workers: conversation extraction (turning a pasted chat into structured memories), the consolidator, and the native assistant’s replies.

Crucially, it does not power retrieval. The search cascade (FTS → vector → graph → rerank) is deterministic and runs without any LLM — a small model can never degrade your memory quality. The model only shapes how well conversations get turned into memories.

Any GGUF model llama.cpp can load is a drop-in replacement — no rebuild needed. Edit ~/.esment/config.toml on the machine and restart the server:

[local_llm]
enabled = true
binary_path = ""   # empty = auto-detect llama-server
model_path = "/Users/ops/models/qwen3-14b-q4_k_m.gguf"
port = 8181
ctx_size = 8192    # raise for longer contexts

Drop the file anywhere on disk and point model_path at it — the engine doesn’t care where it lives. For a fleet, script the same config across machines.

In a truly offline environment, download the GGUF on any connected machine, carry it over (USB / internal mirror), and drop it in the models folder or set model_path. After that, the engine needs no internet connection.

Qwen2.5-3B runs on any machine. For deployments with more RAM, these are the tiers worth considering (GGUF q4_k_m):

• 8GB RAM — keep the 3B. Adequate for extraction.
• 16GB RAM — 7–9B class (Qwen3-8B, Gemma-4-9B/12B). Noticeably better extraction.
• 32GB RAM — 14–32B dense or 30B-class MoE (Qwen3-30B-A3B, GLM-5.1-9B/32B). The quality sweet spot.
• 64GB+ RAM — 70B-class or gpt-oss-120b for near-frontier extraction quality.

• Recommended default — Qwen2.5-3B GGUF (q4_k_m, ~2 GB). Put the file in ~/.esment-enterprise/models (by hand on air-gapped machines) and the app picks it up.
• Qwen (Qwen3.5 / Qwen3.6 family) — huggingface.co/Qwen.
• Gemma 4 — huggingface.co/google.
• GLM (GLM-5.1) — huggingface.co/zai-org.
• gpt-oss (20B / 120B) — huggingface.co/openai.
• Anything else — search quantized builds: Hugging Face GGUF search.

Always prefer the model’s official repository over third-party re-uploads — official repos publish audited quantizations and the license/card for your compliance review.

Honest expectations: Qwen2.5-3B is a small model — in the same league as other sub-4B models of its generation. For Esment’s narrow tasks (extract facts and preferences from a conversation, summarize, classify) it’s competent and reliable; it is not in the same class as frontier models for open-ended reasoning, long-horizon planning, or complex code.

Quantization (how much a model is compressed) q3_k_m versus q4_k_m costs a few percent on benchmarks — far less than the gap between model sizes. If extraction accuracy matters to your deployment, the cheapest win is moving up a size class (7–12B), not chasing the highest quantization of the 3B.