Semantic search and vector databases

Chunking strategies, FAISS and Chroma indexes, hybrid keyword plus vector search, and reranking with a cross-encoder.

Chunking decides the ceiling

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,          # characters, not tokens: roughly 200 tokens
    chunk_overlap=120,       # about 15% overlap keeps a sentence from being cut
    separators=["

", "
", ". ", " ", ""],
    length_function=len,
)

chunks = splitter.split_text(long_document)
print(len(chunks), [len(c) for c in chunks[:5]])

# split on structure first when the document has headings
import re
sections = re.split(r"
(?=#{1,3} )", markdown_doc)      # keep the heading attached
prepared = [c.metadata | {"section": c.page_content.splitlines()[0]}
            for c in splitter.create_documents(sections)]

• Chunk size trades recall against precision: small chunks match queries precisely but lose the context needed to answer; large chunks answer well but match loosely.
• Overlap prevents a sentence at a boundary from being unrecoverable. Ten to twenty percent of the chunk size is the usual range.
• Attach metadata to every chunk — document id, section heading, date, source URL — and inject it into the embedded text. Metadata is what makes filtering and citation possible later.
• Prepend the document title to each chunk before embedding. It costs a few tokens and measurably improves retrieval on documents with generic body text.

Building and querying an index

import numpy as np
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("all-MiniLM-L6-v2")
vectors = encoder.encode(chunks, normalize_embeddings=True, batch_size=64)
vectors = np.asarray(vectors, dtype="float32")

# FAISS: exact search is fine up to about a million vectors
import faiss
index = faiss.IndexFlatIP(vectors.shape[1])       # inner product on unit vectors
index.add(vectors)
print(index.ntotal)

query = encoder.encode(["how do I cancel my subscription?"], normalize_embeddings=True)
scores, ids = index.search(np.asarray(query, dtype="float32"), k=5)
for score, idx in zip(scores[0], ids[0]):
    print(round(float(score), 3), chunks[idx][:90])

# approximate search for scale
ivf = faiss.IndexIVFFlat(faiss.IndexFlatIP(384), 384, 256, faiss.METRIC_INNER_PRODUCT)
ivf.train(vectors)
ivf.add(vectors)
ivf.nprobe = 16                                   # higher is more accurate and slower
print(ivf.search(np.asarray(query, dtype="float32"), 5)[1])

# Chroma: a persistent store with metadata filtering built in
import chromadb
client = chromadb.PersistentClient(path="./chroma")
collection = client.get_or_create_collection("docs", metadata={"hnsw:space": "cosine"})
collection.add(
    ids=[str(i) for i in range(len(chunks))],
    embeddings=vectors.tolist(),
    documents=chunks,
    metadatas=[{"source": "handbook", "section": "billing"} for _ in chunks],
)
print(collection.query(query_embeddings=query.tolist(), n_results=5,
                       where={"section": "billing"})["documents"][0][:1])

Hybrid search and reranking

from rank_bm25 import BM25Okapi

corpus_tokens = [c.lower().split() for c in chunks]
bm25 = BM25Okapi(corpus_tokens)

def hybrid(query, k=10, alpha=0.5):
    lexical = np.asarray(bm25.get_scores(query.lower().split()))
    lexical = lexical / (lexical.max() + 1e-9)

    q = encoder.encode([query], normalize_embeddings=True).astype("float32")
    dense_scores, dense_ids = index.search(q, len(chunks))
    dense = np.zeros(len(chunks))
    dense[dense_ids[0]] = (dense_scores[0] + 1) / 2

    combined = alpha * dense + (1 - alpha) * lexical
    order = np.argsort(combined)[::-1][:k]
    return [(int(i), float(combined[i])) for i in order]

# rerank the top candidates with a cross-encoder, which sees query and passage together
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

candidates = [chunks[i] for i, _ in hybrid("cancel subscription", k=20)]
pairs = [("cancel subscription", c) for c in candidates]
scores = reranker.predict(pairs)
best = np.argsort(scores)[::-1][:5]
print([(round(float(scores[i]), 3), candidates[i][:60]) for i in best])

• Bi-encoders embed query and document independently, so they are fast enough to search millions of vectors and comparatively imprecise.
• Cross-encoders score a query and a passage together and are far more accurate, but they cost one forward pass per candidate. Retrieve 20-100, rerank, keep 3-5.
• Hybrid search wins on exact identifiers, product codes and rare names, which dense vectors smooth away. Weight the two by a tuned alpha rather than 50/50 by default.
• Always normalise the two score scales before combining. Raw BM25 scores have an unbounded range and will dominate cosine similarities otherwise.

FAQ

Related

Word and sentence embeddings Question answering and reading comprehension

Last refreshed 2026-09-18.