Ollama
The easiest way to get a local model answering on your own machine. One command installs it, one pulls a model, and an API appears on localhost that most tools already speak.
Tested on Llama 3.3 70B ·
CLI · Free, open source
Ollama does one thing well: it hides the parts of llama.cpp you did not want to learn. Pull a model, run it, and an OpenAI-compatible endpoint is waiting on port 11434. Most editor plugins and scripts can point at it unchanged.
What it does well. Model management is simple. Switching from a 3B to a 70B model takes one command, and it handles the memory mapping. Modelfiles bake a system prompt and parameters into a named model, so your rules live in the model and not in every script that calls it.
What it does badly. The defaults are conservative and say nothing about it. The context window stays smaller than the model supports until you set it, and people lose hours wondering why a long document gets cut off. Concurrency is limited, which makes it a workstation tool and a poor way to serve a team. GPU support beyond CUDA and Metal is still patchy.
What I would use instead. vLLM to serve several people at once. LM Studio to click around and compare models without a terminal. For one developer who wants a local model reachable from scripts, this.
Notes from the author
Set num_ctx explicitly in a Modelfile before you conclude a model is bad at long input. The default is often far below what the model supports and it truncates silently.
For information only, not professional advice. You use it at your own risk and we are not liable for any loss or damage. See the disclaimer.
Tagged