Information theory for machine learning

Entropy, cross-entropy, KL divergence, mutual information and perplexity — what they measure and which loss to pick for which output.

Entropy and surprise

Entropy measures the average surprise of a distribution: H(p) = -sum p log p, in nats with the natural log or bits with log base 2. A certain outcome has zero entropy; a uniform distribution over k outcomes has maximum entropy log k.

import numpy as np

def entropy(p, eps=1e-12):
    p = np.asarray(p, dtype=float)
    p = p / p.sum()
    return float(-(p * np.log(p + eps)).sum())

entropy([1.0, 0.0, 0.0])                 # 0.0  no surprise
entropy([1/3, 1/3, 1/3])                 # 1.0986 = log(3)
entropy([0.7, 0.2, 0.1])                 # 0.8018

def perplexity(p):
    return float(np.exp(entropy(p)))

perplexity([0.25] * 4)                   # 4.0 — "as uncertain as choosing among 4"

Perplexity is the effective number of equally likely choices: a language model with perplexity 20 is as uncertain as rolling a fair 20-sided die per token.
Lower perplexity is not automatically better for a task. A model can be confidently wrong, and a well-calibrated model with slightly higher perplexity can generate better text.
Entropy of the label distribution is the irreducible floor for a classifier. If your accuracy is near (1 - H) / ... intuition says the labels are noisy, not that the model is bad.
Always add a small epsilon before log: a single zero probability makes the loss infinite.

Cross-entropy and KL divergence

def cross_entropy(p, q, eps=1e-12):
    """p = true distribution, q = predicted probabilities."""
    p, q = np.asarray(p, float), np.asarray(q, float)
    return float(-(p * np.log(q + eps)).sum())

def kl(p, q, eps=1e-12):
    p, q = np.asarray(p, float), np.asarray(q, float)
    return float((p * np.log((p + eps) / (q + eps))).sum())

p = np.array([1.0, 0.0, 0.0])            # one-hot label
q = np.array([0.7, 0.2, 0.1])

cross_entropy(p, q)                      # 0.3567 = -log(0.7)
kl(p, q)                                 # identical here: H(p) is 0

# for a general p, H(p, q) = H(p) + KL(p || q)
print(entropy(p) + kl(p, q))             # same number as cross_entropy(p, q)

Minimising cross-entropy over the model's parameters is exactly minimising KL(p || q), because the label entropy H(p) does not depend on the model. That is the whole reason cross-entropy is the standard classification loss: it is the divergence you actually want, minus a constant.

Quantity	Formula	Symmetric?	Use
Entropy `H(p)`	`-sum p log p`	n/a	Uncertainty of one distribution; label noise floor
Cross-entropy `H(p,q)`	`-sum p log q`	No	Classification and language-model loss
KL `KL(p\|\|q)`	`sum p log(p/q)`	No	Variational inference, distillation, RLHF penalty
Jensen-Shannon	symmetric KL mixture	Yes	Comparing two distributions, GAN-style metrics
Mutual information	`H(X) - H(X\|Y)`	Yes	Feature selection, representation learning

Choosing the loss from the output

Prediction	Loss	Framework call	Note
Single class, `k` options	Categorical cross-entropy	`CrossEntropyLoss(logits, target)`	Expects raw logits, not softmax
Binary label	Binary cross-entropy	`BCEWithLogitsLoss`	Use the logits variant: it is stable
Multi-label, overlapping classes	Binary cross-entropy per label	`BCEWithLogitsLoss`	Do not use softmax here
Real value	Mean squared error	`MSELoss`	Assumes Gaussian noise
Real value with outliers	Huber / smooth L1	`SmoothL1Loss`	Quadratic near zero, linear far out
Distribution over a vocabulary	Cross-entropy over tokens	`CrossEntropyLoss`	Perplexity is its exponential

# the numerically stable way to combine log-softmax and negative log-likelihood
logits = np.array([[2.0, 1.0, 0.1]])
target = np.array([0])

def log_softmax(z):
    z = z - z.max(axis=-1, keepdims=True)
    return z - np.log(np.exp(z).sum(axis=-1, keepdims=True))

loss = -log_softmax(logits)[np.arange(len(target)), target].mean()
print(round(float(loss), 4))             # 0.4170

⚠️

Never compute softmax and then log in separate steps for a loss. Log-softmax-then-NLL is stable; softmax-then-log underflows to -inf for a confidently correct prediction and produces a NaN gradient.

FAQ

Why is KL divergence not a distance?

It is asymmetric: KL(p||q) and KL(q||p) penalise different mistakes and give different numbers. It is also not a metric, so it violates the triangle inequality. Minimising it is still exactly what maximum likelihood does.

How do I pick between cross-entropy and MSE for classification?

Use cross-entropy. MSE with a sigmoid or softmax output gives a vanishing gradient on confident mistakes, while cross-entropy keeps a linear penalty in the logit error and never saturates in the wrong direction.

Probability and distributions Statistical estimation

Last refreshed 2026-09-18.

Information theory for machine learning

Entropy and surprise

Cross-entropy and KL divergence

Choosing the loss from the output

FAQ

Related