Ollama
The simplest way to get a local model answering on your own machine. One command to install, one to pull a model, and an API on localhost that most tools already speak.
Tested on Llama 3.3 70B ·
CLI · Free, open source
Ollama does one thing well: it hides the parts of llama.cpp you did not want to learn. Pull a model, run it, and there is an OpenAI-compatible endpoint on port 11434 that most editor plugins and scripts can point at without changes.
What it does well. Model management is genuinely simple. Switching between a 3B and a 70B model is one command and it handles the memory mapping. Modelfiles let you bake a system prompt and parameters into a named model, which means your rules live in the model instead of in every script that calls it.
What it does badly. The defaults are conservative and quiet about it - the context window is smaller than the model supports unless you set it, and people spend hours wondering why a long document gets truncated. Concurrency is limited, so it is a workstation tool rather than a way to serve a team. GPU support outside CUDA and Metal is still uneven.
What I would use instead. For serving several people at once, vLLM. For clicking around and comparing models without touching a terminal, LM Studio. For one developer who wants a local model reachable from scripts, this.
Notes from the author
Set num_ctx explicitly in a Modelfile before you conclude a model is bad at long input. The default is often far below what the model supports and it truncates silently.
Tagged