Memory and retrieval
Context windows, summarisation, retrieval-augmented generation and what to store outside the model.
The context window is your working memory
A model sees only the tokens in the request. As a conversation grows, cost and latency rise while attention to early detail falls — so context has to be managed, not just accumulated.
| Strategy | When it fits |
|---|---|
| Keep a rolling window | Short chats where old turns stop mattering |
| Summarise older turns | Long sessions; lose detail, keep intent |
| Retrieve on demand (RAG) | Large corpora; pull only what the task needs |
| Store facts in a database | Anything that must be exact: ids, amounts, states |
| Pin the goal | Agents; repeat the objective in every request |
💡
Memory is a product decision, not a parameter. Decide what must be exact (keep in a database), what must be recalled roughly (summarise), and what can be forgotten (drop).
Retrieval-augmented generation
documents ─▶ split into chunks ─▶ embed ─▶ vector store
│
question ─▶ embed ─▶ similarity search ─▶ top-k chunks
│
prompt = question + chunks ─▶ answerhits = index.search(embed(question), k=5)
context = "\n\n".join(h["text"] for h in hits if h["score"] > 0.75)
answer = model(f"""Answer using ONLY the context below.
If the answer is not there, reply "Not found in the provided documents".
Context:
{context}
Question: {question}""")- Chunk by meaning — headings and paragraphs, not fixed character counts.
- Retrieve a handful of chunks, not everything: more context often lowers accuracy.
- Always allow "not found". A model forced to answer will invent.
- Keep the source id with each chunk so answers can cite and you can debug.
State that must live outside the model
Treat the model as stateless. Anything that must survive, be audited, or stay exact belongs in your own storage.
| Data | Where it belongs |
|---|---|
| Conversation history | Your database, trimmed before each call |
| User preferences | Your database, injected as a short summary |
| Business facts | The system of record — queried through tools |
| Long-term knowledge | Vector store or search index |
| Secrets | Never in context; injected only inside tools |
⚠️
Putting credentials or personal data into the context means sending them to a third-party provider on every request, and they may surface in the model's output. Keep secrets in the tool layer, not the prompt.
FAQ
Do I need a vector database?
Not for a few hundred documents — a keyword index or in-memory embeddings are fine. Move to a dedicated store when you need scale, filters or persistence.
Should I fine-tune to add knowledge?
No. Fine-tuning shapes behaviour and format; it is a poor way to store facts. Retrieve facts, fine-tune style.
Related
Tools and function calling Machine learning in one page
Last refreshed 2026-09-18.