Pipelines and preprocessing

Compose preprocessing and a model into one object, keep the test set clean, and handle numeric and categorical columns in a single ColumnTransformer.

Why pipelines

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = ["age", "income", "sessions"]
categorical = ["country", "plan"]

preprocess = ColumnTransformer(
    transformers=[
        ("num", Pipeline([
            ("impute", SimpleImputer(strategy="median")),
            ("scale", StandardScaler()),
        ]), numeric),
        ("cat", Pipeline([
            ("impute", SimpleImputer(strategy="most_frequent")),
            ("encode", OneHotEncoder(handle_unknown="ignore")),
        ]), categorical),
    ],
    remainder="drop",
)

model = Pipeline([
    ("prep", preprocess),
    ("clf", LogisticRegression(max_iter=1000, class_weight="balanced")),
])

model.fit(df_train, y_train)
preds = model.predict(df_test)      # raw DataFrame in, labels out

One object covers imputation, encoding, scaling and the estimator — the exact same transformation is applied at training and serving time.
The pipeline splits the data internally: transformers are fitted on the training fold only, which removes the leak by construction.
handle_unknown="ignore" keeps serving alive when a category appears that was not in training; without it, transform raises.
remainder="drop" is the default; use remainder="passthrough" to keep columns you did not list, or to catch a column you forgot.

Leakage in practice

from sklearn.model_selection import cross_val_score

# WRONG: the scaler saw every row, including the validation folds
X_all_scaled = StandardScaler().fit_transform(X_all)
scores = cross_val_score(LogisticRegression(), X_all_scaled, y, cv=5)
print("too optimistic", scores.mean())

# RIGHT: scaling happens inside each fold
pipe = Pipeline([("scale", StandardScaler()), ("clf", LogisticRegression())])
scores = cross_val_score(pipe, X_all, y, cv=5, scoring="roc_auc")
print("honest", scores.mean())

Transformation	Leak if fitted globally	Why
Scaling / normalising	Yes	Mean and standard deviation encode all rows
Imputation	Yes	The median is a statistic of the whole dataset
One-hot encoding	Mildly	Category vocabulary leaks train/test split
Target encoding	Severely	Directly encodes the label of validation rows
Feature selection	Severely	Picks features that correlate with the test labels
SMOTE / resampling	Severely	Synthetic points from validation rows enter training

⚠️

Resampling for imbalance must happen inside the cross-validation loop. Oversampling before cross_val_score copies rows across folds and can inflate AUC by 20 points or more on a small, imbalanced dataset. Use imblearn.pipeline.Pipeline, which applies samplers only to the training fold.

Custom transformers and feature logic

from sklearn.base import BaseEstimator, TransformerMixin

class RatioFeatures(BaseEstimator, TransformerMixin):
    """Add derived columns; implement fit/transform to plug into any pipeline."""

    def __init__(self, columns=("sessions", "age")):
        self.columns = columns

    def fit(self, X, y=None):
        self.n_features_in_ = X.shape[1]
        return self

    def transform(self, X):
        X = X.copy()
        num, den = self.columns
        X["sessions_per_year"] = X[num] / X[den].clip(lower=1)
        return X

model = Pipeline([
    ("ratio", RatioFeatures()),
    ("prep", preprocess),
    ("clf", LogisticRegression(max_iter=1000)),
])

from sklearn import set_config
set_config(display="diagram")      # renders the pipeline in a notebook

# inspect what the pipeline produced
feature_names = model.named_steps["prep"].get_feature_names_out()
print(len(feature_names))

Because the pipeline is a single estimator, a final fit on all training data is all you need before deployment. Persist the whole object with joblib.dump(model, "model.joblib") — and load it with the same library versions, since pickles are version-sensitive.

FAQ

Do I still need to scale for tree models?

No. Decision trees, random forests and gradient boosting split on thresholds, so monotonic rescaling changes nothing. Scale for linear models, SVMs, k-NN, PCA and anything using distances or gradient descent.

How do I add a step that only applies to training?

Use fit_transform logic inside a transformer, or an imblearn sampler, which is skipped during predict. Never branch on y being present outside a transformer — that is how leakage gets in.

Estimators and fit/predict Model selection and metrics

Last refreshed 2026-09-18.

Pipelines and preprocessing

Why pipelines

Leakage in practice

Custom transformers and feature logic

FAQ

Related