HowToCodePage

Using a model API

Tokens, temperature, system prompts, structured output and cost control — the practical mechanics of calling an LLM.

A call, annotated

from openai import OpenAI
client = OpenAI()                     # reads OPENAI_API_KEY from the environment

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.2,                  # lower = more deterministic
    max_tokens=500,                   # cap the cost of a runaway answer
    messages=[
        {"role": "system", "content": "You are a concise technical editor."},
        {"role": "user",   "content": "Summarise this changelog in 3 bullets."},
    ],
)
print(resp.choices[0].message.content)
  • system sets behaviour and constraints; user carries the task; assistant holds previous replies in a multi-turn conversation.
  • temperature 0–0.3 for extraction and code, higher for brainstorming.
  • max_tokens is your cost brake — not a suggestion to the model about length.
  • Always pass an API key from the environment; never commit it to a repository.

Tokens, cost and latency

Models read and write tokens — roughly 4 characters of English, or ¾ of a word. You pay for input plus output, and longer prompts cost both money and latency on every call.

LeverEffect
Shorter system promptLower cost and latency on every request
Fewer examples (few-shot)Cheaper, may reduce accuracy
Smaller modelMuch cheaper; fine for classification and extraction
StreamingSame cost, much better perceived latency
Caching identical prefixesProvider-dependent discount on repeated context
💡
Cheapest useful pattern: a small model for the bulk of requests, escalating to a large one only when confidence is low or the task is genuinely hard. Route by task, not by habit.

Structured output and reliability

Free text is hard to program against. Ask for JSON, validate it, and retry on failure — never assume the model obeyed the format.

import json

prompt = """Extract the invoice as JSON with keys:
invoice_no (string), total (number), currency (ISO 4217).
Reply with JSON only."""

raw = call_model(prompt)
try:
    data = json.loads(raw)
except json.JSONDecodeError:
    raw = call_model(prompt + "\nYour previous reply was not valid JSON. Try again.")
    data = json.loads(raw)

assert isinstance(data["total"], (int, float))
⚠️
Validate every field, and reject rather than guess when a required value is missing. A pipeline that silently accepts malformed model output fails later, in a place that is much harder to debug.

FAQ

Should I use a system prompt or put instructions in the user message?
System prompts for role, tone and non-negotiable rules; user messages for the task and data. Instructions the model must always follow belong in the system prompt.
How do I stop the model inventing facts?
Give it the source text and instruct it to answer only from that, allow "I don't know", and verify key claims. Retrieval plus grounding beats wishful prompting.

The agent loop

Last refreshed 2026-09-18.