Probability and distributions
Random variables, the distributions ML actually uses, and how cross-entropy is just negative log likelihood in disguise.
Random variables
import numpy as np
rng = np.random.default_rng(0)
# Bernoulli: one coin flip per sample
flips = rng.random(1000) < 0.3
flips.mean() # about 0.3
# Categorical: one outcome from k options
draws = rng.choice(["cat", "dog", "bird"], size=1000, p=[0.5, 0.3, 0.2])
# Gaussian: continuous noise
noise = rng.normal(loc=0.0, scale=1.0, size=1000)
noise.mean(), noise.std() # close to 0.0 and 1.0def softmax(logits):
z = logits - logits.max(axis=-1, keepdims=True) # stability
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
logits = np.array([2.0, 1.0, 0.1])
softmax(logits) # [0.6590, 0.2424, 0.0986] — sums to 1| Distribution | Outcome | Where ML uses it |
|---|---|---|
| Bernoulli | Yes / no | Sigmoid output, binary label |
| Categorical | One of k classes | Softmax output, next-token prediction |
| Gaussian | A real number | Regression noise, VAEs, weight init |
| Uniform | Any value in a range | Random init, sampling temperature |
| Poisson | Count of events | Rate modelling, count data |
⚠️
Always subtract the maximum logit before
exp. Plain np.exp([800, 801]) overflows to inf and the division yields nan; np.exp([-1, 0]) gives the identical probabilities safely. Frameworks do this internally, but hand-written softmax is a common source of silent NaN losses.Conditional probability and likelihood
# P(A|B): among emails that contain "invoice", how many are spam?
b = mask_with_word & mask_spam
p_a_given_b = b.sum() / mask_with_word.sum()
# Bayes: flip a conditional using the base rate
# P(spam|word) = P(word|spam) * P(spam) / P(word)
p_word_given_spam = 0.60
p_spam = 0.20
p_word = 0.15
p_spam_given_word = p_word_given_spam * p_spam / p_word # 0.80A model trained with maximum likelihood maximises the probability it assigns to the observed labels. Because products of many small probabilities underflow, you maximise the sum of log probabilities instead — which is the negative of the cross-entropy loss you minimise in every training loop.
# cross-entropy for one example: -log(probability of the true class)
probs = np.array([0.70, 0.20, 0.10])
true_index = 0
loss = -np.log(probs[true_index]) # 0.3567
# the same loss, written for a batch
def cross_entropy(probs, labels):
picked = probs[np.arange(len(labels)), labels]
return -np.log(picked + 1e-12).mean()Expectation, variance and sampling
x = rng.normal(5.0, 2.0, size=100_000)
x.mean() # expectation: the long-run average
x.var() # variance: mean squared distance from the mean
x.std() # same units as x
# the standard error shrinks with the square root of the sample size
sem = x.std() / np.sqrt(len(x)) # about 0.006 here- Expectation is linear:
E[aX + bY] = aE[X] + bE[Y], always. This is why averaging losses over a batch is valid. - Variance is not linear:
Var(X + Y) = Var(X) + Var(Y)only when X and Y are independent. Correlated errors do not cancel. - Monte Carlo estimates converge as
1 / sqrt(n)— ten times the samples buys you about three times the precision. - A model's softmax score is a probability only if it is calibrated. On its own it is a confidence number, and modern networks are frequently overconfident.
FAQ
Why is cross-entropy better than accuracy as a training loss?
Accuracy is a step function — moving a prediction from 0.51 to 0.99 changes nothing, so the gradient is zero almost everywhere. Cross-entropy is smooth and punishes confident mistakes heavily, giving the optimiser a direction at every step.
What does a temperature setting actually do?
It divides the logits before the softmax. Lower temperature sharpens the distribution toward the top choice; higher temperature flattens it, increasing variety at the cost of accuracy.
Related
Vectors and matrices for ML Evaluation and overfitting
Last refreshed 2026-09-18.