Evaluation and overfitting
Cross-validation, the precision/recall trade-off, and how to spot a model that memorised the training set.
Overfitting, underfitting, just right
| Symptom | Train score | Validation score | Fix |
|---|---|---|---|
| Overfitting | Very high | Clearly lower | More data, simpler model, regularisation, early stopping |
| Underfitting | Low | Low | Richer features, more capacity, longer training |
| Leakage | Suspiciously perfect | Falls apart live | Remove target-derived features; split by entity/time |
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import GradientBoostingClassifier
scores = cross_val_score(
GradientBoostingClassifier(), X_train, y_train,
cv=5, scoring="roc_auc")
print(scores.mean(), scores.std()) # report the spread, not just the meanA high mean with a large standard deviation means the result depends on which fold you got — the model is not stable yet, and the headline number is luck.
Metrics that match the cost
| Metric | Answers | Use when |
|---|---|---|
| Accuracy | Share correct | Balanced classes only |
| Precision | Of predicted positives, how many are right | False positives cost money or trust |
| Recall | Of real positives, how many found | Missing one is dangerous (fraud, disease) |
| F1 | Harmonic mean of both | You need one balanced number |
| ROC AUC | Ranking quality across thresholds | Comparing models before picking a threshold |
| MAE / RMSE | Typical error size | Regression; RMSE punishes big misses |
from sklearn.metrics import precision_recall_curve
prec, rec, thresholds = precision_recall_curve(y_test, proba)
# choose the operating point from the business cost, then hard-code it
best = thresholds[(prec > 0.9)].min()⚠️
A model outputs a probability; the threshold is a business decision. 0.5 is a default, not a recommendation — tune it against the real cost of each error type.
Two traps worth naming
- Class imbalance: resample, use
class_weight="balanced", or change the metric. Never "fix" it by deleting the majority class. - Temporal drift: evaluate on the most recent period, not a random split, or you will ship a model that is already out of date.
- Group leakage: if one customer contributes many rows, split by customer.
- Calibration: probabilities may rank correctly but be badly scaled; check with a reliability curve before using them as probabilities.
💡
Report a confidence interval or the cross-validation spread. A single number from one split invites over-confidence in an evaluation that was itself random.
FAQ
How many folds for cross-validation?
Five is a sensible default; ten when data is scarce. For time-series use
TimeSeriesSplit so each fold trains only on the past.My model scores 0.99. Is that good?
It is suspicious. Check for leakage — an id, a post-outcome field, or duplicated rows split across train and test — before believing it.
Related
Machine learning in one page Pipelines and saving models
Last refreshed 2026-09-18.