Running your first local model

1. Work out your memory budget first

   At 4-bit quantisation a model needs roughly 0.6 GB per billion parameters, plus one to two GB for the context. So an 8B model needs about 5 GB and a 70B needs about 42 GB. On a Mac the number that matters is unified memory; on a PC it is VRAM, and spilling into system RAM makes it slow enough to abandon.

2. Install Ollama

   One command on Linux, an installer on macOS and Windows. It runs as a background service and listens on port 11434.

   curl -fsSL https://ollama.com/install.sh | sh

3. Pull a model that fits

   Start smaller than your budget allows. A model that answers in two seconds gets used; one that takes thirty gets abandoned regardless of how much better it is.

   ollama pull llama3.2:8b
   ollama run llama3.2:8b "Explain what a context window is in two sentences."

4. Set the context window explicitly

   This is the step everyone skips and then concludes local models are bad at long documents. The default is far below what the model supports. Put your parameters and system prompt in a Modelfile so they travel with the model.

   # Modelfile
   FROM llama3.2:8b
   
   PARAMETER num_ctx 32768
   PARAMETER temperature 0.3
   
   SYSTEM """You answer in plain English. When you are unsure, say so in one sentence rather than hedging throughout."""
   
   # then:
   # ollama create mine -f Modelfile
   # ollama run mine

5. Point your existing tools at it

   The endpoint speaks the OpenAI API, so most scripts and editor plugins work by changing the base URL and passing any string as the key.

   curl http://localhost:11434/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "mine",
       "messages": [{"role": "user", "content": "One sentence on why quantisation loses quality."}]
     }'

Notes from the author

Generation slower than about five tokens a second means the model is not fully in memory. Drop to a smaller model or a lower quantisation instead of waiting. Quality on extraction and drafting changes far less than speed does.

For information only, not professional advice. You use it at your own risk and we are not liable for any loss or damage. See the disclaimer.