Word and sentence embeddings

Why dense vectors replaced sparse counts, how cosine similarity works, and how to use sentence embeddings for real retrieval.

From sparse counts to dense vectors

A one-hot vector gives every word the same distance from every other word, so cat and dog look as unrelated as cat and carburettor. Embeddings fix this by learning coordinates where words used in similar contexts end up close together.

import numpy as np

# Sparse: similarity is always zero until two documents share a literal term
one_hot = np.eye(3)                 # rows: cat, dog, car
print(float(one_hot[0] @ one_hot[1]))     # 0.0 - no notion of similarity

# Dense: distance encodes relatedness learned from co-occurrence
cat, dog, car = (np.array(v) for v in ([0.9, 0.1, 0.0], [0.8, 0.2, 0.1], [0.0, 0.9, 0.8]))

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

print(round(cosine(cat, dog), 3))   # 0.99 - related
print(round(cosine(cat, car), 3))   # 0.12 - unrelated

Representation	Context aware	Good for
One-hot / bag of words	No	Teaching the concept; nothing else
TF-IDF	No	A strong sparse baseline for search and classification
Static embeddings (Word2Vec, GloVe, fastText)	No - one vector per word	Word similarity, clustering, small custom models
Contextual embeddings (BERT, sentence transformers)	Yes	Retrieval, classification, anything where meaning depends on the sentence

Using embeddings in practice

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")
docs = [
    "Reset your password from the account settings page.",
    "Billing is charged monthly on the signup date.",
    "Password reset links expire after one hour.",
]

# normalise_embeddings makes the dot product equal to cosine similarity
emb = model.encode(docs, normalize_embeddings=True)
query = model.encode("I cannot log in to my account", normalize_embeddings=True)

scores = emb @ query                       # one number per document
for score, doc in sorted(zip(scores, docs), reverse=True):
    print(round(float(score), 3), doc)

Normalise both sides before comparing, or rank by dot product only if the model was trained that way.
Compare a sentence with a sentence, not with a single word - sentence models are trained on sentence-length input.
Similarity is relative. A score of 0.6 means nothing on its own; look at the gap between the top hit and the rest.
Embed once and store the vectors. Re-embedding a corpus on every query is the most common performance mistake in retrieval.

💡

Static embeddings give one vector per word, so bank in a river and bank in a mortgage share a single ambiguous point. Contextual models produce a different vector for each occurrence, which is why they dominate retrieval today. Either way, the embedding encodes similarity, not truth - it is a search tool, not a fact.

FAQ

Do the famous analogies (king minus man plus woman equals queen) still hold?

Partially, and less reliably than the demo suggests. They depend on the corpus and often reflect stereotyped co-occurrence rather than clean logic. Use them as an intuition, not as evidence of reasoning.

Which embedding model should I start with?

A small sentence-transformer such as all-MiniLM-L6-v2 for prototyping, then evaluate a larger multilingual model on your own queries. Measure retrieval quality on real queries before you switch.

Text preprocessing and tokens Retrieval-augmented generation

Last refreshed 2026-09-18.

Word and sentence embeddings

From sparse counts to dense vectors

Using embeddings in practice

FAQ

Related