Regularisation and normalisation

Dropout, weight penalties, batch versus layer normalisation, augmentation, and early stopping — what each one actually does to the training signal.

Dropout and weight penalties

import tensorflow as tf

l2 = tf.keras.regularizers.l2(1e-4)

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(128,)),
    tf.keras.layers.Dense(256, activation="relu", kernel_regularizer=l2),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(128, activation="relu", kernel_regularizer=l2),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(1),
])

# regularisation loss is part of the reported total; inspect it separately
model.compile(optimizer="adam", loss="mse", metrics=["mae"], run_eagerly=False)
print(model.losses)      # the penalty terms, empty until a forward pass has run

Dropout only activates when training=True. It is applied automatically inside fit and disabled inside predict and evaluate — a manual model(x) call defaults to inference, which is a common source of confusing results.
Dropout on the input layer is a form of noise injection and often helps on tabular data; on image models, augmentation is the better tool.
L2 pushes all weights toward zero proportionally; L1 pushes many weights exactly to zero. Use L1 only when you want sparsity, since it makes the loss non-smooth.
Regularisation loss is not shown in the metric column unless you ask for it. A gap between your tracked loss and the reported total is usually the penalty.

BatchNormalization versus LayerNormalization

# batch norm: statistics over the batch, per channel
bn_model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(28, 28, 1)),
    tf.keras.layers.Conv2D(32, 3, padding="same"),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Activation("relu"),
])

# layer norm: statistics per example, independent of batch size
ln_model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(100, 32)),
    tf.keras.layers.Dense(32),
    tf.keras.layers.LayerNormalization(),
    tf.keras.layers.Activation("gelu"),
])

# batch norm behaviour is genuinely different in eval mode
x = tf.random.normal((8, 28, 28, 1))
train_out = bn_model(x, training=True)
eval_out = bn_model(x, training=False)
tf.reduce_mean(train_out).numpy(), tf.reduce_mean(eval_out).numpy()

Property	BatchNormalization	LayerNormalization
Statistics computed	Across the batch, per channel	Across features, per example
Depends on batch size	Yes, badly at small sizes	No
Works with sequence length changes	Awkward	Yes
Train and eval differ	Yes, uses running stats	No
Common in	Convolutional vision models	Transformers, RNNs, small batches

⚠️

Batch normalisation uses batch statistics while training and running statistics while evaluating. If those running averages are stale — a very short training run, or a batch size that changed between training and inference — the model silently scores worse after export. Check with training=False during validation from the start.

Augmentation and stopping

augment = tf.keras.Sequential([
    tf.keras.layers.RandomFlip("horizontal"),
    tf.keras.layers.RandomRotation(0.05),
    tf.keras.layers.RandomZoom(0.1),
], name="augment")

model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(224, 224, 3)),
    augment,                                  # inside the model: runs only when training
    tf.keras.layers.Rescaling(1.0 / 255),
    tf.keras.layers.Conv2D(16, 3, activation="relu"),
    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(1),
])

callbacks = [
    tf.keras.callbacks.EarlyStopping(
        monitor="val_loss", patience=5, restore_best_weights=True),
    tf.keras.callbacks.ReduceLROnPlateau(
        monitor="val_loss", factor=0.5, patience=2, min_lr=1e-6),
]

history = model.fit(train_ds, validation_data=val_ds, epochs=50, callbacks=callbacks)

Put augmentation inside the model as preprocessing layers. It is then exactly the augmentation used at training time, and it travels with the saved model.
Augment on the training split only. Applying flips and rotations to validation makes the metric meaningless.
Early stopping with restore_best_weights=True is the cheapest regulariser: it costs nothing at training time and removes the need to guess an epoch count.
Always keep an un-augmented validation set as the honest number, and a small training-subset score as a sanity check that the model can fit the data at all.

FAQ

Can I use both dropout and weight decay?

Yes, and they are complementary: dropout perturbs activations while decay shrinks parameters. But tune them one at a time against a fixed validation split, or you will not know which change caused the improvement.

Why does my model perform worse in evaluation than during training?

In order of likelihood: regularisation is active during training and off during evaluation, batch-norm running statistics are stale, or the evaluation pipeline differs from the training pipeline. Compare a training-set evaluation against a validation evaluation to separate them.

Layers, activations and custom models Transfer learning and fine-tuning

Last refreshed 2026-09-18.

Regularisation and normalisation

Dropout and weight penalties

BatchNormalization versus LayerNormalization

Augmentation and stopping

FAQ

Related