Running a model on your own machine
What local models are genuinely good for, what hardware you need, and what to change in prompts that came from a hosted model.
Local models are worth setting up for two reasons: data that cannot leave your machine, and work you run often enough that per-token pricing adds up. For everything else, a hosted model is faster and better.
Hardware, honestly
The number that decides everything is memory. A 4-bit model needs roughly 0.6 GB per billion parameters, plus room for the context.
A 7 to 8 billion parameter model runs on 8 GB and is useful for extraction, classification and short drafting. A 14B model wants 12 to 16 GB and is where local coding assistance starts being pleasant. A 32B model needs 24 GB or more and gets close to a mid-tier hosted model on many tasks.
Apple Silicon shares memory between CPU and GPU, so a 32 GB Mac runs models that would need an expensive discrete card otherwise. On PC, VRAM is the constraint - system RAM works but is slow enough to change how you use it.
Getting started
Ollama is the shortest path: install it, ollama pull qwen2.5-coder:14b, ollama run. LM Studio gives the same thing with a UI and easier parameter tweaking.
Then set your parameters in a Modelfile rather than repeating them. The worked example in the configs section covers the settings that matter.
The setting everyone misses
Ollama defaults to a small context window regardless of what the model supports. Files longer than that are silently truncated, and the model gets blamed for missing what it never saw. Set num_ctx explicitly to what your memory allows.
What to change in your prompts
A prompt written for Claude or GPT will usually work, but less reliably.
Split multi-part instructions. Four constraints in one prompt becomes two prompts with two each. Simplify the output format - JSON with three flat keys is fine, deeply nested JSON is not. Drop the clever framing; smaller models follow direct instructions better than elaborate role-play. Keep temperature low, around 0.2, because on a quantised model randomness shows up as invented API calls rather than as style.
Then test on a task you already know the answer to. A local model that is wrong in a way you cannot detect is worse than no local model.
Tagged