Inference
Running a trained model to produce output - what happens every time you send a prompt.
Training builds the model; inference uses it. Every request you send is an inference run, and its cost is dominated by how many tokens go in and come out.
Two phases matter for latency. Prefill processes your whole prompt at once and is fast but grows with prompt length. Decode generates the answer one token at a time and sets how long you wait for a long response. That is why time-to-first-token and total time behave differently, and why streaming makes an answer feel faster without being faster.
Tagged
Where this comes up
Prompts, configs and tutorials in the library that touch this term.
- ConfigLlamaMistral
Ollama Modelfile for a local code assistant
A Modelfile with a system prompt and parameters tuned for short, correct code answers on a local model.
- GuideLlamaMistral
Running a model on your own machine
What local models are genuinely good for, what hardware you need, and what to change in prompts that came from a hosted model.
- ToolLlamaAPPFree for personal use
LM Studio
A desktop app for downloading, running and comparing local models without a terminal. Best tool for deciding which quantisation is good enough before…
Tested on Llama 3.2 8B · May 2026
- CheatsheetLlama
Ollama commands worth knowing
The subset of the CLI you actually use, plus the two flags that fix the problems everybody hits.
Tested on Llama 3.3 70B · May 2026
- GuideUniversal
Choosing a model for the job
A practical comparison of what to reach for when, based on the shape of the task rather than benchmark scores.
- ToolLlamaMistralCLIFree, open source
Ollama
The simplest way to get a local model answering on your own machine. One command to install, one to pull a model, and an API on localhost that most…
Tested on Llama 3.3 70B · May 2026