Feature engineering and text features

OneHotEncoder, OrdinalEncoder, KBinsDiscretizer, PolynomialFeatures, and turning raw text into a matrix with CountVectorizer and TfidfVectorizer.

Encoding categories and numbers

Transformer	Output	Use when
`OneHotEncoder`	One binary column per category	Nominal categories, low cardinality
`OrdinalEncoder`	One integer column	Genuinely ordered categories, or tree models
`KBinsDiscretizer`	Bin index per value	Non-linear effects for a linear model
`PolynomialFeatures`	Products and powers	Interactions a linear model must be told about
`FunctionTransformer`	Anything you write	Log, clip, or a custom column operation

from sklearn.preprocessing import (FunctionTransformer, KBinsDiscretizer,
                                   OneHotEncoder, OrdinalEncoder, PolynomialFeatures)

ohe = OneHotEncoder(handle_unknown="ignore", min_frequency=5,
                    sparse_output=False)
ordinal = OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1)
bins = KBinsDiscretizer(n_bins=8, encode="onehot-dense", strategy="quantile")

# a log transform as a pipeline step, no custom class needed
log1p = FunctionTransformer(np.log1p, inverse_func=np.expm1, validate=True)

handle_unknown="ignore" and a matching unknown_value keep serving alive when a category appears that training never saw.
min_frequency groups rare categories into a single bucket, which fights the curse of a wide, sparse matrix.
Discretisation creates steps in the prediction: think about whether a jump at a bin edge is acceptable for the decision being made.

Text as features

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.pipeline import Pipeline

tfidf = TfidfVectorizer(
    lowercase=True, ngram_range=(1, 2), min_df=3, max_df=0.9,
    sublinear_tf=True, strip_accents="unicode", stop_words="english")

text_pipe = Pipeline([("tfidf", tfidf), ("clf", LogisticRegression(max_iter=1000))])
text_pipe.fit(train_texts, y_train)

names = text_pipe.named_steps["tfidf"].get_feature_names_out()
coefs = text_pipe.named_steps["clf"].coef_[0]
top = np.argsort(coefs)[-8:][::-1]
print([(names[i], round(float(coefs[i]), 3)) for i in top])

Counts answer 'how often does this word appear'; TF-IDF down-weights terms that appear in almost every document and up-weights the distinctive ones. For a linear model on text, TF-IDF with word and bigram features and a bit of regularisation is a baseline that is genuinely hard to beat.

Interactions and the cost of width

poly = PolynomialFeatures(degree=2, interaction_only=True,
                         include_bias=False)
X_poly = poly.fit_transform(X[["age", "income"]])
print(X.shape[1], "->", X_poly.shape[1])

# bound the blow-up: only interact a chosen subset
interactions = ColumnTransformer([
    ("poly", PolynomialFeatures(degree=2, interaction_only=True,
                                include_bias=False), ["age", "income"]),
    ("rest", "passthrough", ["sessions"]),
])

⚠️

PolynomialFeatures(degree=2) on a hundred columns produces over five thousand, and degree 3 produces hundreds of thousands. Each one costs memory, training time and a fresh chance to overfit, so choose interactions deliberately and always inside a pipeline so the expansion is fitted per fold.

FAQ

One-hot or ordinal encoding for a tree model?

Modern gradient-boosting implementations split on the raw category values efficiently, so ordinal encoding is often enough and keeps the matrix narrow. One-hot remains the safe choice for linear models and SVM.

How do I handle text and numbers together?

Use a ColumnTransformer with a TfidfVectorizer on the text column and the usual numeric pipeline on the rest, then combine the matrices and fit one estimator on the result.

Missing data and outliers Linear models

Last refreshed 2026-09-18.

Feature engineering and text features

Encoding categories and numbers

Text as features

Interactions and the cost of width

FAQ

Related