Serving Ollama in Docker and over a network

Container images, GPU passthrough, volumes for models, OLLAMA_HOST binding, reverse proxies, and access control that is not optional.

Running the container

# CPU only, models on a named volume
docker run -d --name ollama \
  -p 127.0.0.1:11434:11434 \
  -v ollama-models:/root/.ollama \
  -e OLLAMA_KEEP_ALIVE=30m \
  -e OLLAMA_NUM_PARALLEL=2 \
  --restart unless-stopped \
  ollama/ollama:0.5.4

# NVIDIA GPU passthrough
docker run -d --gpus=all --name ollama-gpu \
  -p 127.0.0.1:11434:11434 \
  -v ollama-models:/root/.ollama \
  ollama/ollama:0.5.4

# pull models inside the running container
docker exec -it ollama ollama pull llama3.2:3b-instruct-q4_K_M
docker exec -it ollama ollama list

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:0.5.4
    ports:
      - "127.0.0.1:11434:11434"        # bind to loopback, not 0.0.0.0
    volumes:
      - ollama-models:/root/.ollama
    environment:
      OLLAMA_KEEP_ALIVE: "30m"
      OLLAMA_NUM_PARALLEL: "2"
      OLLAMA_MAX_LOADED_MODELS: "1"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  ollama-models:

• Pin the image tag. ollama/ollama:latest changes under you, and a behaviour change in the server is very hard to attribute.
• Mount a named volume for /root/.ollama. Without it, every container recreation re-downloads tens of gigabytes of weights.
• Bind to 127.0.0.1 by default. Ollama has no authentication, so publishing the port is publishing an open model endpoint.
• GPU passthrough needs the NVIDIA container toolkit on the host. If ollama ps shows a CPU split, the GPU is not being used and the container cannot see it.

Binding and access control

# bind to all interfaces, then protect it with a reverse proxy
OLLAMA_HOST=0.0.0.0:11434 ollama serve

# Caddyfile — TLS plus a bearer token check
ollama.internal.example.com {
    @unauthenticated not header Authorization "Bearer {env.OLLAMA_TOKEN}"
    respond @unauthenticated "unauthorized" 401

    reverse_proxy 127.0.0.1:11434 {
        flush_interval -1          # required for streaming responses
        transport http {
            read_timeout 600s
            write_timeout 600s
        }
    }
}

A client that handles a remote server

import os
import time
import requests

BASE = os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434")
TOKEN = os.environ.get("OLLAMA_TOKEN")
HEADERS = {"Authorization": f"Bearer {TOKEN}"} if TOKEN else {}

def ready(timeout=120, interval=2):
    """Wait for the server, including a model pull in progress."""
    deadline = time.time() + timeout
    while time.time() < deadline:
        try:
            r = requests.get(f"{BASE}/api/tags", headers=HEADERS, timeout=5)
            if r.status_code == 200:
                return True
            if r.status_code == 401:
                raise SystemExit("authentication failed: check OLLAMA_TOKEN")
        except requests.ConnectionError:
            pass
        time.sleep(interval)
    return False

def chat(prompt, model="llama3.2:3b-instruct-q4_K_M", retries=3):
    payload = {"model": model, "prompt": prompt, "stream": False,
               "options": {"num_ctx": 4096, "num_predict": 256}}
    for attempt in range(retries):
        try:
            response = requests.post(f"{BASE}/api/generate", json=payload,
                                     headers=HEADERS, timeout=300)
            response.raise_for_status()
            return response.json()["response"]
        except (requests.ConnectionError, requests.Timeout) as exc:
            if attempt == retries - 1:
                raise
            time.sleep(2 ** attempt)          # exponential backoff
    return None

if not ready():
    raise SystemExit(f"ollama at {BASE} did not become ready")
print(chat("Name one reason to run a model locally."))

• Health-check the server before serving traffic. A container that is up but still loading a model will fail the first requests.
• Retry connection errors with exponential backoff; a server under load or reloading a model refuses connections briefly.
• Send the token on every request. A proxy rule that relies on the client remembering to authenticate is a rule that will be bypassed eventually.
• Set the request timeout above your slowest expected generation. A 300-second timeout is normal for a long completion on CPU.

FAQ

Related

Running models locally Performance tuning

Last refreshed 2026-09-18.