Running your first local model
Ollama on a normal laptop, from install to a model answering in about ten minutes, plus how to work out which size actually fits before you download 40 GB.
Tested on Llama 3.2 8B ·
-
Work out your memory budget first
At 4-bit quantisation a model needs roughly 0.6 GB per billion parameters, plus one to two GB for the context. So an 8B model needs about 5 GB and a 70B needs about 42 GB. On a Mac the number that matters is unified memory; on a PC it is VRAM, and spilling into system RAM makes it slow enough to abandon.
-
Install Ollama
One command on Linux, an installer on macOS and Windows. It runs as a background service and listens on port 11434.
Step 2 curl -fsSL https://ollama.com/install.sh | sh -
Pull a model that fits
Start smaller than your budget allows. A model that answers in two seconds gets used; one that takes thirty gets abandoned regardless of how much better it is.
Step 3 ollama pull llama3.2:8b ollama run llama3.2:8b "Explain what a context window is in two sentences." -
Set the context window explicitly
This is the step everyone skips and then concludes local models are bad at long documents. The default is far below what the model supports. Put your parameters and system prompt in a Modelfile so they travel with the model.
Step 4 # Modelfile FROM llama3.2:8b PARAMETER num_ctx 32768 PARAMETER temperature 0.3 SYSTEM """You answer in plain English. When you are unsure, say so in one sentence rather than hedging throughout.""" # then: # ollama create mine -f Modelfile # ollama run mine -
Point your existing tools at it
The endpoint speaks the OpenAI API, so most scripts and editor plugins work by changing the base URL and passing any string as the key.
Step 5 curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "mine", "messages": [{"role": "user", "content": "One sentence on why quantisation loses quality."}] }'
Everything here assumes you have never run a model locally. The only thing you really need to understand first is memory, because it decides which models are available to you and nothing else matters until that is settled.
Notes from the author
If generation is slower than about five tokens a second, the model is not fully in memory. Drop to a smaller model or a lower quantisation rather than waiting it out - the quality difference on extraction and drafting is much smaller than the speed difference.
Tagged