Gradients and calculus intuition

Derivatives as sensitivity, the chain rule behind backpropagation, and how to tell a learning-rate problem from an architecture problem.

Derivatives measure sensitivity

The derivative of a function tells you how much the output moves when you nudge the input. For a model with millions of parameters, the gradient is the vector of those sensitivities — it says which knob to turn and in which direction to reduce the loss.

def f(x):
    return 3 * x ** 2          # df/dx = 6x

h = 1e-5
numeric = (f(2.0 + h) - f(2.0 - h)) / (2 * h)    # central difference
analytic = 6 * 2.0
numeric, analytic            # 12.000000000006, 12.0

# a gradient check for a small function of two variables
def g(v):
    x, y = v
    return x ** 2 + 3 * x * y + y ** 2      # grad = (2x + 3y, 3x + 2y)

def numeric_grad(fn, v, h=1e-5):
    grad = []
    for i in range(len(v)):
        plus, minus = list(v), list(v)
        plus[i] += h
        minus[i] -= h
        grad.append((fn(plus) - fn(minus)) / (2 * h))
    return grad

point = (1.0, 2.0)
numeric_grad(g, point)        # [8.0, 7.0]
[2 * point[0] + 3 * point[1], 3 * point[0] + 2 * point[1]]   # [8.0, 7.0]

💡

Use a gradient check only as a debugging tool: finite differences cost one extra function evaluation per parameter, so they are hopeless for a real network but perfect for confirming that a hand-written layer's backward pass matches its forward pass.

The chain rule and gradient descent

# one parameter, one step, written out by hand
w = 0.0
lr = 0.1
xs = np.array([1.0, 2.0, 3.0, 4.0])
ys = 2.0 * xs + 1.0                  # true relation: y = 2x + 1

for step in range(20):
    pred = w * xs                    # forward
    loss = ((pred - ys) ** 2).mean() # MSE
    grad = (2 * (pred - ys) * xs).mean()   # d loss / d w, via the chain rule
    w -= lr * grad                   # step downhill
    if step % 5 == 0:
        print(step, round(w, 4), round(loss, 4))

Backpropagation is the chain rule applied from the loss backwards: each layer multiplies the gradient it receives by its own local derivative.
The gradient points uphill, so the update subtracts it. Getting that sign wrong makes the loss grow without bound.
Learning rate is the single most important hyperparameter: it multiplies every gradient, and it interacts with batch size (doubling the batch roughly halves gradient noise, so it tolerates a larger rate).
Local minima are usually less of a problem than saddle points in high dimensions; momentum and adaptive optimisers exist to keep progress through both.

Reading the loss curve

Symptom	Likely cause	First thing to try
Loss becomes NaN or inf	Exploding gradients, bad learning rate	Lower the rate; add gradient clipping
Loss flat from step 0	Rate too low, or learning-rate schedule	Raise the rate by 10x
Loss oscillates or diverges	Rate too high	Lower it, or warm up over a few hundred steps
Loss falls then plateaus early	Underfitting or no capacity	Bigger model, more features, fewer regularisers
Train loss falls, validation rises	Overfitting	Regularisation, augmentation, early stopping, more data
Train and validation both stuck high	Data or label problem	Inspect the inputs and labels before the model

# clip the gradient norm so a single bad batch cannot destroy training
import numpy as np

def clip(grad, max_norm=1.0):
    norm = np.linalg.norm(grad)
    if norm > max_norm:
        grad = grad * (max_norm / norm)
    return grad

# and always compare against a trivial baseline before trusting a score
# e.g. predict the majority class, or the mean of y for regression

Plot the loss, not just the final number. A curve tells you whether you have a step-size problem (zigzag or divergence), a capacity problem (plateau while validation is still falling), or a data problem (both curves stuck from the first epoch).

FAQ

Do I need to derive gradients by hand to use a framework?

No — autodiff computes them. But understanding the chain rule is what lets you interpret vanishing or exploding gradients, choose an initialisation, and debug a layer whose backward pass does not match its forward pass.

What is a good starting learning rate?

For Adam-family optimisers on a small model, 1e-3 is a reasonable default; for SGD with momentum on vision models, 1e-2 with a schedule works. Always confirm with a short run and a loss curve rather than assuming.

Probability and distributions The training loop

Last refreshed 2026-09-18.

Gradients and calculus intuition

Derivatives measure sensitivity

The chain rule and gradient descent

Reading the loss curve

FAQ

Related