Choosing and evaluating models

How to pick a local model by task rather than by leaderboard, and how to test it against your own data.

Pick by task, not by hype

Task	What to optimise for	Typical choice
Classification / routing	Speed, low cost	Small model (1–3B)
Extraction to JSON	Instruction-following, format discipline	Small–mid, low temperature
Summarisation	Coherence, length control	Mid (7–8B)
Code edits	Trained on code	A code-specialised model
Reasoning / planning	Accuracy on multi-step problems	Largest you can run
Embeddings	Retrieval quality, dimension	A dedicated embedding model

💡

Use a big model to design and a small model to serve. Prototype with the strongest model you have, then distil the task into a prompt a smaller one can follow.

Evaluate on your own inputs

Public benchmarks measure general ability on someone else's data. What matters is whether the model does your job acceptably at your latency budget.

cases = [
    ("Refund my £40 order #A-1042", "refund"),
    ("How do I reset my password?", "how_to"),
    ("Your app deleted my files!!", "escalate"),
]

for model in ["llama3.2", "qwen2.5:7b", "phi4"]:
    ok = 0
    for text, expected in cases:
        got = ask(f"Classify as one of refund|how_to|escalate:\n{text}", model=model).strip()
        ok += (got == expected)
    print(model, f"{ok}/{len(cases)}")

Build a small labelled set (50–200 examples) from real traffic.
Measure task accuracy and tokens/second — a 3% accuracy gain for 5× the latency may not be worth it.
Record failure cases verbatim; they become your prompt fixes and your regression tests.

Routing and fallback

def answer(prompt):
    out = ask(prompt, model="llama3.2", temperature=0.1)
    if not is_confident(out):            # your own check, e.g. missing fields
        out = ask(prompt, model="qwen2.5:14b")   # escalate
    return out

⚠️

Always define behaviour for "the local model is unavailable" — a queued job, a cached answer, or an explicit error. Silently returning nothing is the worst option.

FAQ

Which model should I start with?

A current 7–8B general instruct model with q4_K_M quantisation. It runs on a typical developer laptop and is good enough to learn the whole pipeline before you optimise.

Do quantised models hurt quality?

Moderately, and mostly on reasoning and rare tokens. Compare on your own eval set — for classification and extraction, q4 is often indistinguishable at a fraction of the memory.

Calling Ollama from code Evaluation and overfitting

Last refreshed 2026-09-18.

Choosing and evaluating models

Pick by task, not by hype

Evaluate on your own inputs

Routing and fallback

FAQ

Related