Splitting data and cross-validation strategies

train_test_split in detail, KFold, StratifiedKFold, GroupKFold, TimeSeriesSplit, and repeated evaluation that reports a spread instead of one lucky number.

train_test_split in detail

import numpy as np
from sklearn.model_selection import train_test_split

X_dev, X_test, y_dev, y_test = train_test_split(
    X, y,
    test_size=0.2,          # or train_size=0.8, never both
    random_state=42,        # makes the split reproducible
    stratify=y,             # preserves the class ratio in both parts
    shuffle=True)           # must be False for ordered data

print(np.bincount(y_test))         # check the class balance survived
print(X_dev.shape, X_test.shape)

Argument	Effect	When to change it
`test_size`	Fraction held out	Smaller when data is scarce and CV will cover you
`random_state`	Fixes the shuffle	Always while developing
`stratify=y`	Keeps class ratios	Any classification problem
`shuffle=False`	Keeps original order	Time series and ordered logs
`groups=`	Sent to the splitter, not used here	When entities contribute several rows

train_test_split only partitions rows. It cannot know that twenty rows belong to the same customer or that row 5,000 is next week's data — you must encode that structure yourself, or use a splitter that does.

Choosing a cross-validation splitter

from sklearn.model_selection import (GroupKFold, KFold, StratifiedGroupKFold,
                                     StratifiedKFold, TimeSeriesSplit, cross_val_score)

cv_strat = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_plain = KFold(n_splits=5, shuffle=True, random_state=42)
cv_group = GroupKFold(n_splits=5)
cv_time = TimeSeriesSplit(n_splits=5, gap=1)
cv_sgroup = StratifiedGroupKFold(n_splits=5)

for tr, va in cv_group.split(X, y, groups=customer_id):
    print(len(tr), len(va))

# every splitter plugs into the same evaluation helpers
scores = cross_val_score(pipe, X, y, cv=cv_strat, scoring="roc_auc", n_jobs=-1)

Splitter	Splits by	Use when
`KFold`	Random rows	Regression, or classification with balanced classes
`StratifiedKFold`	Random rows, class ratio preserved	Classification — the default choice
`GroupKFold`	Entity groups	Several rows per customer, patient or device
`StratifiedGroupKFold`	Groups plus class ratio	Grouped classification with rare positives
`TimeSeriesSplit`	Time windows, train before test	Anything with a temporal signal
`RepeatedStratifiedKFold`	Several shuffles of StratifiedKFold	Small datasets; reduces split luck

# a pipeline is what makes the score honest: preprocessing refits in every fold
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([("scale", StandardScaler()),
                 ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))])

Reporting the result

from sklearn.model_selection import RepeatedStratifiedKFold, cross_validate

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=42)

res = cross_validate(pipe, X_dev, y_dev, cv=cv,
                     scoring=["roc_auc", "average_precision", "recall"],
                     return_train_score=True, n_jobs=-1)

for key in ["test_roc_auc", "test_average_precision", "test_recall"]:
    s = res[key]
    print(key, round(s.mean(), 4), "+/-", round(s.std(), 4))
print("train-test gap", round(res["train_roc_auc"].mean() - res["test_roc_auc"].mean(), 4))

Quote the mean and the standard deviation. A difference smaller than the spread is not a difference.
A large train-minus-test gap is overfitting; both scores low is underfitting.
Keep the test set untouched until the final report — every extra look turns it into a validation set.
Use return_estimator=True if you want to inspect per-fold fits, or return_indices=True to find which rows the model failed on.

⚠️

Pass the raw X to cross_val_score, never a matrix you already scaled, imputed or feature-selected. Those steps learn from every row, so doing them once outside the loop leaks the validation folds and inflates the score.

FAQ

How many folds should I use?

Five is the default; ten for small datasets. More folds cost more fits and only reduce the variance of the estimate, they do not make the model better. Never use more folds than you have rows in the rarest class.

What is the difference between cross_val_score and cross_validate?

cross_val_score returns a single array of scores. cross_validate returns a dictionary with fit and score times, multiple metrics at once, and optionally the train scores and fitted estimators.

Linear models Imbalanced classification

Last refreshed 2026-09-18.

Splitting data and cross-validation strategies

train_test_split in detail

Choosing a cross-validation splitter

Reporting the result

FAQ

Related