Performance tuning

keep_alive, concurrency and model-load limits, context and output caps, GPU offload, and measuring tokens per second instead of guessing.

Server-side settings

# environment variables read by the ollama server at startup
OLLAMA_KEEP_ALIVE=30m          # keep a model resident for 30 minutes after last use
OLLAMA_NUM_PARALLEL=4          # concurrent requests per model
OLLAMA_MAX_LOADED_MODELS=2     # how many models may be resident at once
OLLAMA_MAX_QUEUE=512           # requests queued before rejecting new ones
OLLAMA_NUM_GPU=1               # GPUs to use on a multi-GPU host
OLLAMA_FLASH_ATTENTION=1       # lower KV-cache memory on supported hardware
OLLAMA_HOST=127.0.0.1:11434    # bind address

ollama serve

• OLLAMA_KEEP_ALIVE=0 unloads immediately after every request, which is the right setting for a laptop and the wrong one for a server.
• Parallelism is not free: each concurrent request needs its own KV cache. Raising OLLAMA_NUM_PARALLEL on a machine with just enough memory causes offloading and a slowdown under load.
• Total throughput usually falls as concurrency rises past the point where memory is saturated. More parallel requests give lower latency for more users, not more total tokens per second.
• Apply these on the server process, not in the client. A client cannot raise the concurrency the server allows.

Per-request controls

import requests
import time

BASE = "http://localhost:11434"

def run(prompt, model="llama3.2:3b-instruct-q4_K_M", **options):
    options.setdefault("num_ctx", 4096)
    options.setdefault("num_predict", 256)
    start = time.perf_counter()
    payload = {"model": model, "prompt": prompt, "stream": False,
               "keep_alive": "20m", "options": options}
    data = requests.post(f"{BASE}/api/generate", json=payload, timeout=300).json()

    seconds = time.perf_counter() - start
    eval_count = data.get("eval_count", 0)
    eval_duration = data.get("eval_duration", 1) / 1e9
    return {
        "text": data["response"],
        "seconds": round(seconds, 2),
        "generated_tokens": eval_count,
        "tokens_per_second": round(eval_count / max(eval_duration, 1e-6), 1),
        "prompt_tokens": data.get("prompt_eval_count", 0),
        "load_ms": round(data.get("load_duration", 0) / 1e6, 1),
    }

print(run("Summarise the benefits of caching in three bullets."))
print(run("Summarise the benefits of caching in three bullets.",
          num_ctx=2048, num_predict=128))

• num_predict caps generated tokens. Without it, a chatty model can generate far more than the answer needs and dominate your latency budget.
• num_ctx sets the context window and therefore the KV-cache size. Asking for 32k context when your prompts are 2k wastes memory and slows attention.
• The response includes load_duration: a large value means the model was reloaded, so your keep_alive is too short for your traffic pattern.
• keep_alive can be set per request, which is useful for a nightly batch job that should release memory when it finishes.

Measuring and interpreting

import json
import statistics
import time

import requests

BASE = "http://localhost:11434"

def benchmark(prompts, model="llama3.2:3b-instruct-q4_K_M", repeats=3):
    rates, first_token = [], []
    for _ in range(repeats):
        for prompt in prompts:
            payload = {"model": model, "prompt": prompt, "stream": True,
                       "options": {"num_ctx": 4096, "num_predict": 128}}
            with requests.post(f"{BASE}/api/generate", json=payload,
                               stream=True, timeout=300) as r:
                generated = 0
                start = None
                loaded = 0.0
                for line in r.iter_lines():
                    if not line:
                        continue
                    chunk = json.loads(line)
                    now = time.perf_counter()
                    if start is None:
                        start = now
                        loaded = chunk.get("load_duration", 0) / 1e9
                    if chunk.get("response"):
                        generated += 1
                        if generated == 1:
                            first_token.append(now - start)
                    if chunk.get("done"):
                        total = now - start
                        rates.append(generated / max(total - loaded, 1e-6))
    return {
        "median_tokens_per_second": round(statistics.median(rates), 1),
        "median_first_token_ms": round(statistics.median(first_token) * 1000, 1),
        "samples": len(rates),
    }

print(benchmark(["Write one sentence about caching."], repeats=2))

• Always warm up before measuring. The first call after a load is not representative and will make every configuration look bad.
• Track time to first token separately from tokens per second. Streaming makes the first number the user-facing one and the second the throughput one.
• Compare configurations at the same context length and output length. A shorter num_ctx will always look faster because attention costs less.
• If throughput drops when two requests run together, you are memory-bound and the fix is a smaller model or quantisation, not more parallelism.

FAQ

Related

Running models locally Troubleshooting and the limits of local models

Last refreshed 2026-09-18.