Quantisation
Storing a model with lower-precision numbers so it fits in less memory and runs on ordinary hardware.
Model weights are normally 16-bit floats. Quantisation stores them in 8, 5 or 4 bits, cutting memory roughly in proportion. A model that needs a data-centre GPU at full precision can run on a laptop at 4-bit.
Quality degrades gradually rather than falling off a cliff. At 8-bit the difference is hard to detect; at 4-bit most tasks still work but long instruction lists and precise formatting suffer first, which is exactly what prompts depend on. Below 4-bit the loss is obvious.
This is the main reason a prompt that works on a hosted model needs simplifying for a local one. See the Llama platform page for what to change.
Tagged
Where this comes up
Prompts, configs and tutorials in the library that touch this term.
- CheatsheetLlama
Ollama commands worth knowing
The subset of the CLI you actually use, plus the two flags that fix the problems everybody hits.
Tested on Llama 3.3 70B · May 2026
Ollama
The simplest way to get a local model answering on your own machine. One command to install, one to pull a model, and an API on localhost that most…
Tested on Llama 3.3 70B · May 2026
- Local setupLlama
Running your first local model
Ollama on a normal laptop, from install to a model answering in about ten minutes, plus how to work out which size actually fits before you download…
Tested on Llama 3.2 8B · May 2026
- ConfigLlamaMistral
Ollama Modelfile for a local code assistant
A Modelfile with a system prompt and parameters tuned for short, correct code answers on a local model.
- GuideLlamaMistral
Running a model on your own machine
What local models are genuinely good for, what hardware you need, and what to change in prompts that came from a hosted model.
LM Studio
A desktop app for downloading, running and comparing local models without a terminal. Best tool for deciding which quantisation is good enough before…
Tested on Llama 3.2 8B · May 2026