Model tags, sizes and quantisation explained

The model:tag convention, parameter counts, Q4/Q5/Q8 and FP16 variants, VRAM versus RAM, and pinning tags so a deployment cannot drift.

Reading a model name

# model[:tag] where the tag encodes family, size and quantisation
ollama pull llama3.2:3b-instruct-q4_K_M
ollama pull qwen2.5:7b-instruct-q5_K_M
ollama pull mistral-small:22b-instruct-2409-q4_K_M

# inspect what you actually have
ollama list
ollama show llama3.2:3b-instruct-q4_K_M

# the size answer that matters for capacity planning
ollama ps
# NAME   ID   SIZE   PROCESSOR   UNTIL
# 100% GPU  means fully offloaded; a split percentage means system RAM is in use

• A tag without a quantisation suffix (llama3.2:3b) resolves to a default that the publisher can change. Pin the full tag in production.
• Smaller parameter counts are not merely faster versions of larger ones: they are different models with different failure modes, and a task that a 3B model solves may be impossible for a 1B.
• The size shown by ollama list is the on-disk size. The memory needed at run time is larger, because of the context cache and framework overhead.
• Vision and embedding models are separate families. A chat model does not produce embeddings, and an embedding model cannot answer questions.

Quantisation levels

# compare two quantisations of the same model on the same prompt
ollama run qwen2.5:7b-instruct-q4_K_M "Explain a database index in one sentence."
ollama run qwen2.5:7b-instruct-q8_0  "Explain a database index in one sentence."

# measure throughput rather than guessing
ollama run qwen2.5:7b-instruct-q4_K_M --verbose "Write a 200-word product description."
# the response reports eval rate (tokens/s) and prompt eval rate

• Q4_K_M is the standard recommendation: it is the point where quality loss is small relative to the memory saved. Below Q4 the drop is steep.
• Quantisation hurts reasoning and rare tokens more than it hurts formatting and classification. A 2-bit model can still produce valid JSON while being unable to do arithmetic.
• K-quants (Q4_K_M) use a mixed scheme with more bits for important tensors. The plain legacy Q4_0 is measurably worse at the same size.
• Quantising an already-quantised model compounds the loss. Always quantise from the original FP16 or BF16 weights.

VRAM, RAM and what fits

def memory_needed(params_billion, bits_per_weight, context_tokens,
                   kv_heads=8, head_dim=128, layers=32):
    """Rough planning numbers, not a promise."""
    weights_gb = params_billion * 1e9 * bits_per_weight / 8 / 1e9
    # KV cache: 2 tensors (key and value) per layer
    kv_gb = (2 * layers * kv_heads * head_dim * context_tokens * 2) / 1e9
    overhead_gb = 0.6
    return {"weights": round(weights_gb, 1),
            "kv_cache": round(kv_gb, 2),
            "total": round(weights_gb + kv_gb + overhead_gb, 1)}

print(memory_needed(7, 4.5, 4096))     # a 7B at q4 with a 4k context
print(memory_needed(7, 4.5, 32768))    # the same weights, eight times the context
print(memory_needed(70, 4.5, 8192))    # a 70B needs serious hardware

• Context length has its own memory cost, and it grows linearly. A long-context configuration can need more memory than the weights.
• Weights do not have to fit in VRAM for the model to run, but the layers that spill to system RAM run an order of magnitude slower. Partial offload is a throughput decision, not a solution.
• Two models loaded at once need both sets of weights resident. OLLAMA_MAX_LOADED_MODELS=1 prevents accidental co-residency on a small machine.
• Leave headroom for the display, other processes and the KV cache growth. A model that exactly fits will thrash on the first long conversation.

FAQ

Related

Choosing and evaluating models Performance tuning

Last refreshed 2026-09-18.