Running a model on your own machine

Set up a local model for two reasons: data that cannot leave your machine, and work you run so often that per-token pricing adds up. For everything else a hosted model is faster and better.

Hardware

Memory decides everything. A 4-bit model needs about 0.6 GB per billion parameters, plus room for the context.

A 7 to 8 billion parameter model runs in 8 GB and handles extraction, classification and short drafting. A 14B model wants 12 to 16 GB, and that is where local coding help starts to feel good. A 32B model needs 24 GB or more and gets close to a mid-tier hosted model on many tasks.

Apple Silicon shares memory between CPU and GPU, so a 32 GB Mac runs models that would otherwise need an expensive discrete card. On a PC the limit is VRAM. System RAM works, but slowly enough to change how you use it.

Getting started

Ollama is the shortest path: install it, run ollama pull qwen2.5-coder:14b, then ollama run. LM Studio does the same with a UI and easier parameter changes.

Keep your parameters in a Modelfile so you stop repeating them. The worked example in the configs section covers the ones that matter.

The setting everyone misses

Ollama starts with a small context window whatever the model supports. Longer files are cut off without a warning, and the model gets blamed for missing what it never saw. Set num_ctx to what your memory allows.

What to change in your prompts

A prompt written for Claude or GPT will usually run, with less reliable results.

Split multi-part instructions: four constraints in one prompt become two prompts with two each. Simplify the output. JSON with three flat keys is fine and deeply nested JSON is not. Drop clever framing, since smaller models follow direct instructions better than elaborate role-play. Keep temperature near 0.2, because on a quantised model randomness shows up as invented API calls, not as style.

Then test on a task whose answer you already know. A local model that is wrong in ways you cannot spot is worse than none.