Vectors and matrices for ML

The two objects that every model is built from: vectors as features, matrices as batched linear maps, and the shape rules that decide whether your code runs.

Vectors

A vector is an ordered list of numbers. In ML it is usually one example's features, one embedding, or one row of weights. The operations that matter are the dot product (how aligned two vectors are) and the norm (how long one is).

import numpy as np

a = np.array([1.0, 2.0, 3.0])
b = np.array([4.0, 0.0, 1.0])

a @ b                      # 7.0  dot product = sum(a_i * b_i)
np.linalg.norm(a)          # 3.7417  Euclidean length
a + b                      # elementwise
2 * a                      # scalar broadcast
np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))   # 0.5492  cosine similarity

Operation	Meaning	Where it appears
`a @ b`	Alignment / weighted sum	One neuron's pre-activation
`norm(a)`	Length	Normalising embeddings, gradient clipping
Cosine of two vectors	Alignment ignoring length	Retrieval and similarity search
`a + b`	Elementwise add	Bias terms, residual connections
`a * b`	Elementwise multiply	Gating, attention masks, losses

A dot product is large and positive when the vectors point the same way, zero when they are perpendicular, and negative when opposed.
Cosine similarity divides that by both lengths, so a longer vector does not automatically look more similar — which is exactly what you want in search.
High-dimensional vectors are nearly orthogonal by default: with 768 random dimensions, unrelated items already sit close to zero similarity, so rank by relative score, not by an absolute threshold.

Matrices as linear maps

X = np.array([[1.0, 2.0],
              [3.0, 4.0],
              [5.0, 6.0]])      # shape (3, 2): 3 examples, 2 features

W = np.array([[0.5, -0.5, 1.0],
              [2.0,  0.0, -1.0]])   # shape (2, 3): 2 inputs, 3 outputs

Y = X @ W                       # (3, 2) @ (2, 3) -> (3, 3)
Y.shape                         # (3, 3)

X.T.shape                       # (2, 3)  transpose swaps the axes
np.eye(3)                       # identity: the map that changes nothing

# a batch of inputs through a layer, with bias broadcast over the batch
bias = np.array([0.1, 0.2, 0.3])
Z = X @ W + bias                # (3, 3) + (3,) -> (3, 3)

Read a matrix multiply by its inner dimensions: (m, k) @ (k, n) -> (m, n). The shared k must match, and the result keeps the outer two. When NumPy raises a shape error, this rule tells you which of the two arrays to transpose.

Expression	Result shape	Reading
`(n,) @ (n,)`	scalar	Similarity of two vectors
`(m, k) @ (k,)`	`(m,)`	One example per row -> one output each
`(m, k) @ (k, n)`	`(m, n)`	A whole batch through one layer
`(k, m) @ (m, k)`	`(k, k)`	Gram matrix: all pairwise similarities
`(m, k) @ (m, k)`	error	Inner dimensions do not match

⚠️

Batches go in rows, features in columns — (n_samples, n_features). Every mainstream library (scikit-learn, PyTorch, TensorFlow, Keras) assumes this, and axis errors are the single most common shape bug. If you build (n_features, n_samples) by accident, everything still runs and your model learns nothing useful.

Counting dimensions

# parameter count of a small dense network
def dense_params(n_in, n_out):
    return n_in * n_out + n_out     # weights + biases

dense_params(784, 128)    # 100480
dense_params(128, 10)     # 1290

# memory for a float32 embedding table
rows, dims = 50_000, 768
rows * dims * 4 / 1e6     # 153.6 MB just for the vectors

Parameter count explains model size and much of the training cost — a wider layer is quadratic in cost, a deeper stack is linear.
The same maths scales from one example to a batch, which is why GPUs help: a larger m costs almost nothing extra per row.
Reducing 768 dimensions to 128 with a matrix keeps the pipeline shape-valid while training the projection itself.

FAQ

Do I need calculus before linear algebra?

No. Start with vectors, dot products and matrix multiplication — you can build and train useful models using only those. Gradients become necessary when you want to understand why training works or debug it.

Why normalise vectors before comparing them?

Cosine similarity ignores magnitude, so it compares direction only. That is usually what you want for text and image embeddings, where vector length often reflects document length or token count rather than meaning.

Probability and distributions NumPy arrays

Last refreshed 2026-09-18.

Vectors and matrices for ML

Vectors

Matrices as linear maps

Counting dimensions

FAQ

Related