Imbalanced data

Why accuracy is the wrong headline, resampling and class weights, synthetic oversampling done safely, threshold tuning, and evaluating with precision-recall.

Why accuracy lies

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, average_precision_score

print(np.bincount(y))                       # e.g. [9900, 100]

base = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("accuracy", accuracy_score(y_test, base.predict(X_test)))    # 0.99
print("average precision", average_precision_score(y_test, np.zeros_like(y_test)))

The class balance is a property of the problem, not a defect in the data. Removing negative rows to 'fix' it throws away information the model needs to locate the decision boundary.

Weights, resampling and synthesis

from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.pipeline import Pipeline as ImbPipeline
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

pipe = ImbPipeline([
    ("scale", StandardScaler()),
    ("smote", SMOTE(random_state=42, k_neighbors=5)),   # training folds only
    ("clf", LogisticRegression(max_iter=1000)),
])

# class weights need no synthetic rows and no extra library
weighted = LogisticRegression(max_iter=1000, class_weight="balanced")

• class_weight="balanced" multiplies each class loss by its inverse frequency. It is the cheapest first thing to try.
• Undersampling the majority class speeds training but discards real rows; use it only when the majority is genuinely redundant.
• SMOTE interpolates between a positive row and its neighbours. It helps with continuous features and hurts with one-hot columns, where interpolation produces impossible categories.
• imblearn.pipeline.Pipeline applies the sampler inside each fold. Resampling before cross-validation copies synthetic rows across folds and inflates the score.
• Focal loss is available in gradient-boosting and neural libraries; it down-weights easy examples, which matters most at extreme imbalance.

Choose the operating point

from sklearn.metrics import precision_recall_curve

prob = pipe.predict_proba(X_test)[:, 1]
prec, rec, thr = precision_recall_curve(y_test, prob)

# if the team can review 200 cases a week, take the top 200 by probability
k = 200
order = np.argsort(-prob)[:k]
print("precision at k", y_test.values[order].mean())
print("recall at k", y_test.values[order].sum() / y_test.sum())

FAQ

Related

Classification metrics in depth Splitting data correctly

Last refreshed 2026-09-18.