Interpretability, bias and fairness

Permutation importance, partial dependence, SHAP, subgroup metrics, where bias enters, and documenting what a model must not be used for.

Global and local explanations

Technique	Answers	Cost	Caveat
Coefficients	Direction and size of a linear effect	Free	Only valid for linear models, and only after scaling
Impurity importance	Which features the trees split on	Free	Biased toward high-cardinality features
Permutation importance	How much the metric drops when a column is shuffled	One pass per column	Correlated features split the credit
Partial dependence	Average prediction as one feature varies	Grid of predictions	Assumes features are independent
SHAP values	Per-row attribution that adds up	Expensive on large data	Explains the model, not the world

from sklearn.inspection import permutation_importance

perm = permutation_importance(
    boost, X_val, y_val, scoring="average_precision",
    n_repeats=10, random_state=42, n_jobs=-1)

order = np.argsort(-perm.importances_mean)
for i in order[:8]:
    print(X_val.columns[i], round(perm.importances_mean[i], 4),
          round(perm.importances_std[i], 4))

Subgroup metrics

from sklearn.metrics import average_precision_score, recall_score

rows = []
for name, mask in subgroups.items():
    y_true, y_pred = y_test[mask], (prob[mask] > threshold)
    rows.append({
        "group": name,
        "n": int(mask.sum()),
        "positives": int(y_true.sum()),
        "recall": round(recall_score(y_true, y_pred, zero_division=0), 3),
        "ap": round(average_precision_score(y_true, prob[mask]), 3),
    })
print(pd.DataFrame(rows).to_string(index=False))

Compare recall and precision across groups, not just the headline metric — a strong overall score can hide a group the model barely serves.
Small groups have noisy metrics; report the counts alongside every rate, or you will chase noise.
Bias enters through the label definition, the sampling, the proxy features, and the historical decisions recorded in the data.
A feature can be fair in isolation and still act as a proxy: postcode encodes ethnicity, device model encodes income.

Document limits before someone else discovers them

Intended use: the decision the model supports and who reviews it.
Out-of-scope: the uses it must not be put to — eligibility, discipline, medical triage.
Data: source, period, coverage, known gaps.
Metrics: overall and per subgroup, with the folds and the seed.
Failure modes: the segments where it under-performs and the drift it is sensitive to.
Owner and review date: who answers for it, and when it is re-examined.

⚠️

An explanation is not a justification. SHAP values show which inputs moved a prediction inside the model; they do not show that the decision is right, lawful or fair. Never present an attribution chart as evidence that a decision is defensible.

FAQ

Should I use SHAP or permutation importance?

Permutation importance for a quick global ranking on tabular data, SHAP when you need per-row explanations to support a decision or an appeal. Both describe the model, and both mislead when features are strongly correlated.

Can I remove a sensitive feature to make a model fair?

Not by itself. Correlated features still carry the signal, so measure subgroup outcomes after removal. Fairness is a property of the decisions and their impact, not of the column list.

Deploying and monitoring a model Regression metrics and residual analysis

Last refreshed 2026-09-18.

Interpretability, bias and fairness

Global and local explanations

Subgroup metrics

Document limits before someone else discovers them

FAQ

Related