Retrieval-augmented generation

Chunk and index documents, retrieve the right passages, and answer with citations while allowing an honest not-found reply.

Build the index once

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,          # characters, not tokens - measure with your embedding model
    chunk_overlap=120,       # keep enough context that a sentence is not cut in half
    separators=["\n\n", "\n", ". ", " "],
)
chunks = splitter.split_documents(load_documents("docs/"))

store = Chroma.from_documents(chunks, OpenAIEmbeddings(model="text-embedding-3-small"))
retriever = store.as_retriever(search_kwargs={"k": 4})

Split on structure - headings, paragraphs, list items - not on a fixed character count.
Carry metadata through the splitter: source path, section title, last-updated date, permissions.
Index a small, honest sample and run real queries before you embed ten thousand documents.
Filter by metadata at query time so a user never retrieves a document they are not allowed to read.

Ask with the retrieved context

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer using only the context below. If the answer is not present, "
               "reply exactly: Not found in the provided documents."),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])

def format_docs(docs):
    return "\n\n".join(f"[{d.metadata['source']}] {d.page_content}" for d in docs)

rag = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt | llm | StrOutputParser()
)

print(rag.invoke("How long is the warranty on the A2 controller?"))

Failure	Symptom	Fix
Retrieval miss	The answer exists but never reaches the prompt	Improve chunking, add keyword search, re-rank the top 50 down to 4
Right document, wrong passage	Quoted text is off topic	Smaller chunks with overlap; keep a heading with its body
Unsupported answer	Model answers from its own memory	Force a not-found reply and require a citation for every claim
Stale index	Documents changed, answers did not	Version the index and re-embed on write or on a schedule
Context dilution	Answer ignores the relevant chunk	Retrieve fewer, better chunks and put the question after the context

⚠️

RAG fails quietly. A wrong answer with a confident citation looks identical to a right one, so build the evaluation before you build the interface: a set of real questions with known answers, plus questions whose correct answer is not in the corpus at all.

FAQ

How many chunks should I retrieve?

Start with four and tune against your evaluation set. More chunks raise cost and latency and often reduce accuracy because the relevant passage gets lost among near-misses. Re-ranking beats simply retrieving more.

Should I use a vector store for a few hundred documents?

Not necessarily. In-memory embeddings or a keyword index will do, and they are easier to debug. Move to a dedicated vector database when you need persistence, metadata filters or scale.

Word and sentence embeddings Prompt templates and chains

Last refreshed 2026-09-18.

Retrieval-augmented generation

Build the index once

Ask with the retrieved context

FAQ

Related