Troubleshooting and the limits of local models

Model not found, out-of-memory and slow loads, context overflow, context rot in small models, and the honest point at which a hosted API is the right answer.

Common failures and their fixes

import requests

BASE = "http://localhost:11434"

def diagnose(model="llama3.2:3b-instruct-q4_K_M"):
    report = {}
    try:
        tags = requests.get(f"{BASE}/api/tags", timeout=5).json()["models"]
    except requests.ConnectionError:
        return {"server": "unreachable"}

    names = {m["name"] for m in tags}
    report["model_present"] = model in names
    if model not in names:
        report["available"] = sorted(names)[:10]
        return report

    loaded = requests.get(f"{BASE}/api/ps", timeout=5).json()["models"]
    entry = next((m for m in loaded if m["name"] == model), None)
    report["loaded"] = bool(entry)
    if entry:
        report["size_vram_gb"] = round(entry.get("size_vram", 0) / 1e9, 2)
        report["fully_on_gpu"] = entry.get("size_vram", 0) >= entry.get("size", 1) * 0.95
        report["expires_at"] = entry.get("expires_at")

    probe = requests.post(f"{BASE}/api/generate", json={
        "model": model, "prompt": "Reply with the single word OK.",
        "stream": False, "options": {"num_ctx": 2048, "num_predict": 8},
    }, timeout=300).json()
    report["probe_reply"] = probe["response"].strip()[:40]
    report["load_seconds"] = round(probe.get("load_duration", 0) / 1e9, 2)
    report["tokens_per_second"] = round(
        probe.get("eval_count", 0) / max(probe.get("eval_duration", 1) / 1e9, 1e-6), 1)
    return report

for key, value in diagnose().items():
    print(f"{key:20s} {value}")

• Check ollama ps before anything else. Most "it is slow" reports are a model that is not fully on the GPU or has just been reloaded.
• A tag mismatch is the most common model not found: llama3.2:3b and llama3.2:3b-instruct-q4_K_M are different names even when they resolve to the same weights.
• A truncated context is silent. The model does not warn that your retrieved passages were dropped; it simply answers from what it saw.
• Log load_duration and eval_duration from every response. Together they separate loading cost from generation cost, which decides the fix.

Where local models let you down

• Context rot: small models degrade badly as the prompt grows. A 3B model can be sharp at 2,000 tokens and useless at 8,000, even when the context window nominally allows more.
• Instruction following: long, multi-constraint instructions are followed inconsistently. Split one complex instruction into several simple calls instead of writing a longer prompt.
• Reasoning: multi-step arithmetic, planning and long chains of inference are where the gap to large hosted models is widest. Do not expect a 7B model to be a reliable calculator.
• Tool reliability: a small model may emit the right tool name with malformed arguments, or describe the call in prose. Bound the loop and validate every argument.
• Knowledge currency: a local model's world knowledge is frozen at its training cut-off. Anything recent must come from retrieval, not from the weights.
• Languages: quality outside English and a few high-resource languages drops sharply, and tokenisation costs more tokens per word.

# a routing policy: local first, escalate on a measurable condition
def route(prompt, local_answer, checks):
    """checks returns True when the local answer is good enough."""
    if all(check(local_answer) for check in checks):
        return {"route": "local", "answer": local_answer}

    if not REMOTE_AVAILABLE:
        return {"route": "local_degraded", "answer": local_answer,
                "warning": "answer may be incomplete"}

    return {"route": "remote", "reason": "local answer failed validation"}

def has_minimum_length(answer, minimum=20):
    return len(answer.split()) >= minimum

def no_uncertainty_markers(answer):
    markers = ["i am not sure", "i cannot", "as an ai", "not in context"]
    return not any(marker in answer.lower() for marker in markers)

print(route("What is our refund policy?", "Refunds take 30 days.", [has_minimum_length]))

Choosing local or hosted

# a hybrid architecture, which is what most real systems end up doing
def answer(question, context):
    # stage 1: the local model handles the common, narrow case cheaply
    local = local_model(question, context)
    local_result = validate(local)

    if local_result["ok"] and local_result["confidence"] > 0.8:
        return {"answer": local_result["answer"], "engine": "local"}

    # stage 2: escalate the hard or uncertain cases
    remote = remote_model(question, context)
    if validate(remote)["ok"]:
        return {"answer": validate(remote)["answer"], "engine": "remote"}

    # stage 3: never leave the caller without an answer
    return {"answer": "I could not answer this from the available documents.",
            "engine": "abstain"}

def validate(result):
    return {"ok": bool(result.get("answer") and len(result["answer"]) > 10),
            "confidence": result.get("score", 0.0),
            "answer": result.get("answer", "")}

The practical pattern is local-first with escalation. Most traffic is routine and a local model handles it at no marginal cost; the long tail is routed to a hosted model where quality matters. Both paths share one interface, one evaluation set and one fallback, so you can move the routing threshold as you learn.

FAQ

Related

Choosing and evaluating models Performance tuning

Last refreshed 2026-09-18.