Missing data and outliers
SimpleImputer and KNNImputer inside a pipeline, indicator features for missingness, and robust scalers that stop outliers from dominating the fit.
Imputing inside the pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import KNNImputer, SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
numeric = Pipeline([
("impute", SimpleImputer(strategy="median", add_indicator=True)),
("scale", StandardScaler()),
])
categorical = Pipeline([
("impute", SimpleImputer(strategy="constant", fill_value="missing")),
])
prep = ColumnTransformer([
("num", numeric, numeric_cols),
("cat", categorical, categorical_cols),
])
# KNNImputer uses similar rows, so fit it on scaled data
knn_step = Pipeline([("scale", StandardScaler()),
("impute", KNNImputer(n_neighbors=5, weights="distance"))])| Strategy | Behaviour | Risk |
|---|---|---|
mean / median | Constant per column | Shrinks variance; hides a systematic gap |
most_frequent | Mode of the column | Fine for categories, blunt for numbers |
constant | A fixed sentinel value | The sentinel may collide with a real value |
KNNImputer | Average of the nearest complete rows | Slow on many columns; assumes distance is meaningful |
IterativeImputer | Models each column from the others | Powerful but slow and easy to leak if fitted globally |
add_indicator=Trueappends a binary flag per imputed column, letting the model distinguish 'unknown' from 'zero'.- Fit the imputer on the training fold only. A median computed over the full table has already seen the test rows.
- Impute before
FeatureUnion-style modelling but after any column-specific filtering you wrote — order matters.
Outliers and robust preprocessing
from sklearn.preprocessing import QuantileTransformer, RobustScaler
# median and IQR instead of mean and standard deviation
robust = RobustScaler(quantile_range=(10.0, 90.0))
X_robust = robust.fit_transform(X_train)
# map any distribution onto a normal or uniform one
quant = QuantileTransformer(output_distribution="normal",
n_quantiles=1000, random_state=42)
X_quant = quant.fit_transform(X_train)StandardScaleruses the mean and standard deviation, both of which a single extreme row can move a great deal.RobustScaleruses the median and the interquartile range instead.QuantileTransformeris monotone, so it fixes skew and remains compatible with a linear model, but it hides the original units.- Tree ensembles are indifferent to monotone scaling, so use clipping rather than rescaling when the model is a forest or a boosted model.
- Clip against thresholds derived from training data only, and record them so serving applies exactly the same limits.
Deciding whether the gap matters
Before choosing an imputation strategy, check whether the missingness itself predicts the label. If rows with a missing value have a very different outcome rate, the missingness is information, not noise.
gap = df["income"].isna()
print(df.groupby(gap)["churned"].mean()) # does the gap predict the label?
print(df.isna().mean().sort_values(ascending=False).head()) # how much is missing
print(df[gap].shape[0] / len(df)) # dropping it is only an option when small⚠️
Do not fill missing values with a sentinel that is a plausible real value — a missing age becoming 0, or a missing price becoming 0, teaches the model something false. Use an indicator column plus a neutral imputation, or a sentinel that cannot occur in the data.
FAQ
Should I drop rows with missing values?
Only if the affected rows are few, missing at random, and their removal does not shift the label rate. Otherwise impute within the pipeline and keep the indicator so the model can use the fact that the value was absent.
Which imputer should I use first?
SimpleImputer(strategy='median') with add_indicator=True. Move to KNNImputer or IterativeImputer only if the extra accuracy on a validation split justifies the much higher fitting cost.Related
Feature engineering and text features Clustering and manifold learning
Last refreshed 2026-09-18.