Tree-based models

Decision trees and their pruning knobs, random forests, HistGradientBoostingClassifier, and why impurity-based feature importances mislead.

A single tree

from sklearn.tree import DecisionTreeClassifier, export_text

tree = DecisionTreeClassifier(
    max_depth=4,             # the main brake on overfitting
    min_samples_leaf=20,     # each leaf must cover real rows
    min_samples_split=40,
    ccp_alpha=0.0,           # cost-complexity pruning, tuned separately
    random_state=42).fit(X_train, y_train)

print(tree.get_depth(), tree.get_n_leaves())
print(export_text(tree, feature_names=list(feature_names), max_depth=3))

A fully grown tree memorises the training set: perfect training accuracy, poor validation accuracy.
Trees need no scaling and handle monotone transforms without complaint.
They cannot extrapolate: a tree predicts within the range of the leaves it saw, so trends beyond the training range flatten out.
A single tree is best used for inspection and communication, rarely as the final model.

Forests and boosting

Model	How it combines trees	Strengths	Costs
`RandomForestClassifier`	Bagging: parallel trees on bootstrap samples	Hard to misconfigure, gives out-of-bag estimates	Large memory, flat predictions on smooth trends
`ExtraTreesClassifier`	Random split thresholds as well	Faster, more decorrelated	Slightly higher bias
`GradientBoostingClassifier`	Sequential trees fitting the residual	Strong accuracy, small trees	Slower, sensitive to learning rate
`HistGradientBoostingClassifier`	Histogram-binned gradient boosting	Fast on large data, native NaN support	Binning discretises features	Best default for tabular problems

from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier)

forest = RandomForestClassifier(
    n_estimators=500, max_features="sqrt", min_samples_leaf=2,
    oob_score=True, n_jobs=-1, random_state=42).fit(X_train, y_train)
print("out of bag", round(forest.oob_score_, 4))

boost = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=600, max_leaf_nodes=31,
    l2_regularization=1.0, early_stopping=True,
    validation_fraction=0.15, random_state=42).fit(X_train, y_train)
print("iterations used", boost.n_iter_)

oob_score_ gives a cheap validation estimate for a forest because each tree was trained without roughly a third of the rows. It is a useful sanity check, not a replacement for a proper held-out evaluation.

Feature importances and their pitfalls

import numpy as np
from sklearn.inspection import permutation_importance

# impurity importance: fast, but biased toward high-cardinality features
print(sorted(zip(feature_names, forest.feature_importances_),
             key=lambda p: -p[1])[:5])

perm = permutation_importance(boost, X_val, y_val,
                              scoring="roc_auc", n_repeats=10,
                              random_state=42, n_jobs=-1)
print(sorted(zip(feature_names, perm.importances_mean),
             key=lambda p: -p[1])[:5])

⚠️

Impurity importance is computed on the training data and rewards features with many possible split points, so a random identifier can outrank a genuinely predictive column. Use it for a first glance only; rank features with permutation importance on held-out data.

FAQ

How many trees should a forest have?

Enough that the validation score stops improving — 300 to 800 is typical. More trees only cost time; they do not cause overfitting the way more depth does.

Why does the boosted model stop early?

Early stopping watches the internal validation split and halts when the loss stops improving for n_iter_no_change rounds. That protects against the overfitting that comes from too many sequential trees.

Linear models Feature engineering and text features

Last refreshed 2026-09-18.

Tree-based models

A single tree

Forests and boosting

Feature importances and their pitfalls

FAQ

Related