Regression metrics and residual analysis
MAE, RMSE, MAPE and their failure modes, quantile loss for asymmetric costs, and how residual plots expose systematic error a single number hides.
Pick the error you actually pay
| Metric | Units | Behaviour | Use when |
|---|---|---|---|
| MAE | Same as target | Average absolute error; robust | Every error costs the same |
| RMSE | Same as target | Punishes large errors hard | Big misses are disproportionately bad |
| MAPE | Percent | Breaks down near zero targets | Scale-free comparison of similar series |
| RMSLE | Log scale | Penalises under-prediction more gently | Targets spanning orders of magnitude |
| R squared | Unitless | Relative to the mean baseline | Explaining variance, comparing to a baseline |
import numpy as np
from sklearn.metrics import (mean_absolute_error, mean_squared_error,
r2_score, root_mean_squared_error)
mae = mean_absolute_error(y_test, pred)
rmse = root_mean_squared_error(y_test, pred)
r2 = r2_score(y_test, pred)
print(mae, rmse, r2)
print("typical error", f"{mae / np.mean(y_test):.1%}") # relative to the mean targetQuote the error in the units of the problem, next to the mean target value. 'RMSE 412' means nothing until the reader knows the average order is 3,000 and the current process is off by 700.
Residuals tell you what the score cannot
resid = y_test - pred
# structure in the residuals means the model is missing something
df = pd.DataFrame({"pred": pred, "resid": resid, "segment": segments})
print(df.groupby("segment")["resid"].agg(["mean", "std", "count"]))
# a funnel shape means the error grows with the prediction
print(df.assign(bucket=pd.qcut(df["pred"], 5)).groupby("bucket")["resid"].mean())
# the biggest misses are worth reading one by one
print(df.reindex(resid.abs().sort_values(ascending=False).index).head(10))- Residuals centred above zero in one segment: the model under-predicts that group consistently.
- Growing spread with the predicted value: consider modelling the log of the target or using a quantile loss.
- A pattern against time: the relationship drifted, and retraining on older data will not fix it.
- The ten worst cases are usually a data problem — a bad join, a stale feature, a rare category — not a modelling one.
When the cost is asymmetric
from sklearn.ensemble import GradientBoostingRegressor
# under-stocking is worse than over-stocking: model a high quantile
high = GradientBoostingRegressor(loss="quantile", alpha=0.9, random_state=42)
high.fit(X_train, y_train)
lower = GradientBoostingRegressor(loss="quantile", alpha=0.1, random_state=42)
lower.fit(X_train, y_train)
band = high.predict(X_test) - lower.predict(X_test) # prediction interval width💡
A point prediction plus a sense of its spread is far more useful than a point prediction alone. Quantile models give you an interval you can act on, and the interval width is itself a signal: wide intervals mean the model is unsure and should defer to a human.
FAQ
Is a higher R squared always better?
No. R squared rises as you add features and is undefined when you compare across different target definitions. Use it against a baseline you define, and prefer MAE or RMSE in the target's own units for communication.
Why is my MAPE enormous?
Almost certainly because some true values are close to zero. MAPE divides by the target, so small denominators explode. Switch to MAE or RMSE, or filter to a minimum target size and say so in the report.
Related
Classification metrics in depth Interpretability, bias and fairness
Last refreshed 2026-09-18.