Data collection, cleaning and feature engineering

Where training rows come from, how to treat missing values and outliers without lying to the model, and how to turn raw columns into signals.

Collect before you model

A model is a summary of the rows you fed it. If those rows are a biased sample — only surviving customers, only completed orders, only one region — the model learns the bias and reports a beautiful score on the same biased test set.

import pandas as pd

df = pd.read_csv("events.csv", parse_dates=["created_at"])
df = df.drop_duplicates(subset=["event_id"])
df = df[df["created_at"] < CUTOFF]          # never train on rows newer than the label

print(df.shape)
print(df.dtypes.to_string())
print(df.isna().mean().sort_values(ascending=False).head(10))
print(df["label"].value_counts(normalize=True))

Record a snapshot identifier (date, table version, query hash) with the export. Six months later, nobody can reproduce a score from a file called final_data_v2.csv.

Missing values and outliers

• Missing completely at random — dropping or imputing with the median is usually harmless.
• Missing at random — the gap depends on other observable columns; impute using those columns, not a global constant.
• Missing not at random — the gap itself carries meaning (no value recorded because the customer refused). Keep a flag; do not impute blindly.
• Outliers are three different things: data errors, rare-but-real events, and the tail you actually care about. Decide which before clipping anything.

import numpy as np

# impute, and keep the fact that it was missing
df["income_missing"] = df["income"].isna().astype(int)
df["income"] = df["income"].fillna(df["income"].median())

# an outlier is not always an error: inspect before you clip
q1, q3 = df["amount"].quantile([0.25, 0.75])
iqr = q3 - q1
lo, hi = q1 - 1.5 * iqr, q3 + 1.5 * iqr
print(df.loc[(df["amount"] < lo) | (df["amount"] > hi), "amount"].describe())

df["amount_clipped"] = df["amount"].clip(lower=lo, upper=hi)   # only if justified

The imputation statistic must be learned on the training split only. Compute it inside a pipeline, never on the full table — a median taken across test rows is a leak.

From columns to features

df["hour"] = df["created_at"].dt.hour
df["dow"] = df["created_at"].dt.dayofweek
df["days_since_signup"] = (df["created_at"] - df["signup_at"]).dt.days
df["amount_log"] = np.log1p(df["amount"])
df["orders_per_day"] = df["orders"] / df["tenure_days"].clip(lower=1)

# frequency encoding is safe: it uses no label
freq = df["city"].value_counts(normalize=True)
df["city_freq"] = df["city"].map(freq).fillna(0.0)

FAQ

Related

Splitting data correctly Supervised algorithms in practice

Last refreshed 2026-09-18.