Classification metrics in depth

Confusion matrices, ROC-AUC versus precision-recall curves, choosing an operating threshold, and checking whether probabilities are calibrated.

Everything starts at the confusion matrix

from sklearn.metrics import classification_report, confusion_matrix

cm = confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = cm.ravel()

precision = tp / (tp + fp)      # of the alarms raised, how many were real
recall = tp / (tp + fn)         # of the real cases, how many were caught
fpr = fp / (fp + tn)            # of the clean cases, how many were wrongly flagged
print(cm, precision, recall, fpr)
print(classification_report(y_test, y_pred, digits=3))

Report the matrix and the two error counts, not just a scalar. Stakeholders can argue about how many false positives they can absorb; almost nobody can argue about an AUC.

ROC-AUC versus precision-recall

from sklearn.metrics import (average_precision_score,
                             precision_recall_curve, roc_auc_score, roc_curve)

prob = clf.predict_proba(X_test)[:, 1]
print("roc auc", roc_auc_score(y_test, prob))
print("average precision", average_precision_score(y_test, prob))

fpr, tpr, thr = roc_curve(y_test, prob)
prec, rec, thr_pr = precision_recall_curve(y_test, prob)

• ROC-AUC uses the true negative rate, so with 1% positives a model can look strong while flagging a large share of the negative class.
• Average precision reacts to the positive class only — it is the metric to quote when positives are rare.
• Both describe ranking, not decisions. You still have to pick a threshold to produce labels.
• Compare a new model against the current one on the same test rows and the same metric, never against a number from a different report.

Thresholds and calibration

import numpy as np
from sklearn.calibration import CalibratedClassifierCV

# pick the threshold from the cost of each error type
cost_fp, cost_fn = 5, 50
expected = cost_fp * (1 - prob) + cost_fn * prob
threshold = np.quantile(prob, 0.10)          # flag the riskiest 10%

# are the probabilities themselves trustworthy?
cal = CalibratedClassifierCV(clf, method="isotonic", cv=5).fit(X_train, y_train)
prob_cal = cal.predict_proba(X_test)[:, 1]
print(prob_cal.mean(), y_test.mean())        # predicted rate vs observed rate

FAQ

Related

Imbalanced data Supervised algorithms in practice

Last refreshed 2026-09-18.