Model selection and metrics

Nested cross-validation, grid and randomised search, and picking a metric that reflects the cost of being wrong rather than the one that flatters the model.

Tuning with cross-validation

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, StratifiedKFold
from scipy.stats import loguniform

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

grid = GridSearchCV(
    estimator=model,                       # the whole pipeline
    param_grid={
        "clf__C": [0.01, 0.1, 1.0, 10.0],  # step name + double underscore + param
        "clf__penalty": ["l2"],
        "prep__num__scale": ["passthrough", StandardScaler()],
    },
    scoring="roc_auc",
    cv=cv,
    n_jobs=-1,
    refit=True,
)
grid.fit(df_train, y_train)

grid.best_params_
grid.best_score_                           # mean CV score of the winner
best = grid.best_estimator_                # already refitted on all of train
grid.cv_results_["std_test_score"]         # how stable that mean is

random = RandomizedSearchCV(model, {
    "clf__C": loguniform(1e-3, 1e2),
    "clf__solver": ["lbfgs", "liblinear"],
}, n_iter=30, scoring="roc_auc", cv=cv, random_state=42)

Parameter names are stepname__paramname. One missed underscore silently tunes nothing and leaves the default in place.
Never read best_score_ as the model's performance: the search chose the best of many scores on the same folds, so it is biased upward. Report a separate held-out score or a nested CV instead.
Randomised search covers a large space with a fixed budget and usually beats a coarse grid for the same number of fits.
Use LogisticRegression(max_iter=1000) inside grids: a convergence warning during a search means the reported score is for an unfinished fit.

Metrics that match the problem

from sklearn.metrics import (classification_report, confusion_matrix,
                             f1_score, precision_recall_curve, roc_auc_score)

y_pred = best.predict(X_test)
y_prob = best.predict_proba(X_test)[:, 1]

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))
print("roc auc", round(roc_auc_score(y_test, y_prob), 3))

precision, recall, thresholds = precision_recall_curve(y_test, y_prob)
# choose the threshold from the business cost, not the default 0.5

Metric	Use when	Watch out for
Accuracy	Classes balanced and errors equally costly	Misleading on a 1:100 imbalance
Precision	False positives are expensive (spam filter)	Can be gamed by predicting the rare class almost never
Recall	False negatives are expensive (disease screening)	Rises trivially if you flag everything
F1	You need one balanced number	Hides which side is failing
ROC AUC	Ranking quality, threshold-free	Optimistic under heavy imbalance
Average precision / PR AUC	Rare positives	Preferred over ROC AUC for < 10% positives
MAE / RMSE	Regression	RMSE penalises outliers much harder

💡

A single accuracy number hides which errors you make. Always print the confusion matrix alongside it, and pick the decision threshold from the relative cost of a false positive versus a false negative rather than accepting the library default of 0.5.

Getting an honest number

from sklearn.model_selection import cross_val_score, train_test_split

# hold out a final test set that the search never sees
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=7)

grid.fit(X_dev, y_dev)
grid.best_estimator_.score(X_test, y_test)      # report this number

# is the gap between folds large? then the model is unstable, not tuned
scores = cross_val_score(grid.best_estimator_, X_dev, y_dev, cv=cv)
print(scores.mean(), scores.std(), scores)

Split by time, not at random, for anything with a temporal signal — a random split lets the model peek at the future.
Group by entity when rows are not independent (several records per user or patient); use GroupKFold.
Report a confidence interval or the fold standard deviation. A 0.02 difference between two models on 200 rows is noise.
Record the exact library versions with the score; a metric without the environment is not reproducible.

FAQ

How many folds should I use?

Five or ten for typical datasets. Use more folds for very small data, and fewer for large data where each fit is expensive. Always shuffle for KFold on ordered data, and use StratifiedKFold for classification.

Should I tune every hyperparameter?

No. Tune the two or three that actually move the metric, decide the rest by default or by domain knowledge. Every extra parameter multiplies the fits and increases the chance of overfitting the validation folds.

Pipelines and preprocessing Evaluation and overfitting

Last refreshed 2026-09-18.

Model selection and metrics

Tuning with cross-validation

Metrics that match the problem

Getting an honest number

FAQ

Related