Splitting data correctly

Train, validation and test roles, stratified, grouped and time-series splits, leakage through preprocessing, and nested cross-validation.

Three sets, three jobs

from sklearn.model_selection import train_test_split

# hold out the test set first and lock it away
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

# then split development data into train and validation inside every experiment
X_train, X_val, y_train, y_val = train_test_split(
    X_dev, y_dev, test_size=0.25, random_state=42, stratify=y_dev)

print(X_train.shape, X_val.shape, X_test.shape)

With little data, skip the fixed validation set and use cross-validation on the development split. What you must not do is tune against the test set — after twenty comparisons it is just another training set with extra steps.

Split by structure, not by row

from sklearn.model_selection import GroupKFold, StratifiedKFold, TimeSeriesSplit

# several rows per customer: no customer may appear on both sides
for tr, va in GroupKFold(n_splits=5).split(X, y, groups=df["customer_id"]):
    ...

# future-only training, for anything with a time signal
for tr, va in TimeSeriesSplit(n_splits=5, gap=1).split(X):
    ...

# keep the class ratio stable in every fold
StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

• Random row splits assume rows are independent. Session logs, patients, and repeated orders violate that assumption, and the model will look far better than it is.
• gap in TimeSeriesSplit leaves a buffer between train and validation, which matters when the label is only known a few days after the event.
• Split by entity or by time before any preprocessing that learns statistics.
• Check that the label rate in every fold is close to the overall rate; a wildly different fold means the split is measuring something else.

Nested cross-validation

If you tune hyperparameters on the same folds you use to report the score, the reported number is optimistic — the search picked the configuration that happened to fit those folds. Nested CV separates the two jobs: an inner loop tunes, an outer loop scores.

from sklearn.model_selection import GridSearchCV, cross_val_score

inner = GridSearchCV(model, {"clf__C": [0.1, 1, 10]}, cv=3, scoring="roc_auc")
outer_scores = cross_val_score(inner, X_dev, y_dev, cv=5, scoring="roc_auc")

print(outer_scores.mean(), outer_scores.std())   # the number you may quote

FAQ

Related

Data collection, cleaning and feature engineering Hyperparameter tuning and experiment tracking

Last refreshed 2026-09-18.