Running models locally
Install Ollama, pull a model, and understand why local inference is a different trade-off rather than a free replacement.
Install and pull
# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh
# then, in any shell
ollama pull llama3.2 # downloads weights, cached for later runs
ollama run llama3.2 # interactive chat
ollama list
ollama ps # what is loaded in memory right now- Models are stored on disk (tens of GB are normal) and loaded into RAM or VRAM when used.
- The first run after a pull is slower while weights load; subsequent runs reuse them.
ollama rm <model>frees disk space.
💡
Local models are private and free per request, but you supply the hardware. That trade-off — no per-token cost, fixed capability ceiling — is the whole decision.
What you need
| Model size | Rough RAM/VRAM | Feels like |
|---|---|---|
| 1–2B | 2–4 GB | Fast; fine for classification and simple extraction |
| 7–8B | 6–10 GB | Usable general assistant; the usual starting point |
| 13–14B | 12–18 GB | Noticeably better reasoning, noticeably slower |
| 30–70B | 24–48 GB+ | Near-hosted quality if you have the hardware |
Quantisation (the q4, q5 suffixes) trades a small amount of quality for large memory savings. q4_K_M is the common default balance.
⚠️
If the model does not fit in VRAM, it spills to system RAM and throughput can drop by an order of magnitude. Check with
ollama ps that the model stays fully on the GPU.Serving more than one request
# allow more parallel requests and keep the model resident
OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=2 OLLAMA_KEEP_ALIVE=30m ollama serveBy default Ollama unloads an idle model after a few minutes; a large model therefore pays the load cost repeatedly. OLLAMA_KEEP_ALIVE trades memory for responsiveness.
FAQ
Local or hosted?
Local when privacy, offline use or per-request cost dominate. Hosted when you need the strongest reasoning or the lowest latency without owning a GPU. Many production systems route between the two.
Why is it so slow on my laptop?
Usually the model does not fit in GPU memory, or you selected a size far beyond your hardware. Try a smaller (or more heavily quantised) model.
Related
Calling Ollama from code Using a model API
Last refreshed 2026-09-18.