HowToCodePage

Multimodal AI in practice

Send images and documents to a vision model, transcribe and synthesise speech, and know where each modality still breaks down.

Images

import base64, pathlib

def describe(path, question):
    b64 = base64.b64encode(pathlib.Path(path).read_bytes()).decode()
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": question},
                {"type": "image_url",
                 "image_url": {"url": f"data:image/jpeg;base64,{b64}", "detail": "high"}},
            ],
        }],
    )
    return resp.choices[0].message.content

print(describe("receipt.jpg", "Extract the merchant, date and total as JSON."))
  • Images are tokenised by area. A high-detail image can cost more than a page of text, so downscale to the smallest size that still answers the question.
  • Rotate and deskew before sending. Orientation errors account for a surprising share of bad extractions.
  • Ask for structured output and validate it. A vision model reading a receipt will occasionally invent a plausible total.

Speech

# speech to text
with open("call.mp3", "rb") as f:
    tr = client.audio.transcriptions.create(model="whisper-1", file=f, language="en")
print(tr.text)

# text to speech
with client.audio.speech.with_streaming_response.create(
        model="tts-1", voice="alloy", input="Your order has shipped.") as response:
    response.stream_to_file("reply.mp3")
TaskWorks wellBreaks on
Speech to textClear single-speaker audioHeavy accents, overlapping speakers, jargon
Text to speechShort, neutral announcementsUnusual names, emotional nuance
Image captioningGeneral scenes and objectsSmall text, precise measurements
Document parsingPrinted single-column pagesHandwriting, complex tables, stamps
Chart readingSimple bar and line chartsFine gridlines, approximate axis values

For documents, a pipeline of OCR plus a text model is often more accurate and much cheaper than asking a vision model to transcribe everything. Use vision for layout understanding and for pages where the structure matters.

Where multimodal still fails

⚠️
Models are fluent about images they cannot actually read, producing confident descriptions of blurry text and wrong numbers from charts. For anything where the digits matter, cross-check with OCR or a deterministic parser and route low-confidence results to a human rather than trusting the model's tone.

FAQ

How should I prepare images before sending them?
Crop to the region of interest, rotate to upright, resize so the smallest text is still legible, and convert to JPEG or PNG. Compression artefacts hurt more than resolution helps in most cases.
Can I send a PDF directly?
Some APIs accept PDFs and extract text and pages server-side. Otherwise render the pages to images yourself, which gives you control over resolution and lets you keep the page number in the prompt for citations.

Using a model API Evaluating AI features

Last refreshed 2026-09-18.