Inference
Running a trained model to get output. It happens every time you send a prompt.
Training builds the model and inference uses it. Each request you send is one inference run, and the cost mostly tracks how many tokens go in and come out.
Two phases set the delay. Prefill reads your whole prompt at once, which is quick but grows with prompt length. Decode writes the answer one token at a time and decides how long you wait for a long reply. Time to first token and total time therefore behave differently, and streaming makes an answer feel faster without making it faster.
Tagged
Where this comes up
Prompts, configs and tutorials in the library that touch this term.
- GuideUniversal
Ways to use AI without paying
The free ways to use AI, where each one runs out, and a simple test for whether free is enough for your task.
- ConfigLlama
Modelfile for a consistent local persona
A Modelfile that fixes the system prompt and sampling parameters of a local model, so the same behaviour holds in every session and you stop retyping…
- Local setupLlama
Running your first local model
Ollama on an ordinary laptop, from install to a model answering in about ten minutes. It also shows how to work out which size fits before you…
Tested on Llama 3.2 8B · May 2026
Groq Cloud
Hosted inference on custom LPU hardware, built for speed. Open-weight models answer at a token rate that makes streaming feel close to instant.
Tested on Llama 3.3 70B · Jun 2026
Together AI
Hosted inference for open-weight models such as Llama, Mixtral and Qwen, plus fine-tuning, at a lower price than most proprietary-model APIs.
Tested on Llama 3.3 70B · Jun 2026
AnythingLLM
A desktop app for chatting with your own documents against a local or hosted model. The retrieval pipeline is built in, so you do not assemble it…
Tested on Llama 3.3 70B · Jun 2026