Supervised algorithms in practice

Linear and logistic regression, k-NN, trees, forests, gradient boosting and SVMs, with the assumptions each one quietly makes about your data.

Start linear

from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge

lin = LinearRegression().fit(X_train_scaled, y_train)
log = LogisticRegression(max_iter=1000).fit(X_train_scaled, y_train_cls)
ridge = Ridge(alpha=1.0).fit(X_train_scaled, y_train)

log.predict_proba(X_test_scaled)[:, 1]     # probabilities
log.coef_                                  # effect per feature, given the scaling
log.classes_                               # column order for predict_proba

A linear model assumes the outcome is a weighted sum of the features — or, for logistic regression, that the log-odds are. This is often surprisingly hard to beat, trains in milliseconds, and produces coefficients you can defend in a meeting. Regularisation (alpha, C) is how you trade bias for stability.

What each family assumes

Algorithm	Assumes	Good at	Watch out for
Linear / logistic	Additive effect, roughly linear log-odds	Interpretable baselines, sparse text data	Unscaled features, non-linear structure
k-NN	Nearby points share labels	Small data, smooth boundaries	Curse of dimensionality, needs scaling, slow at predict
Decision tree	Axis-aligned splits are enough	Interaction-heavy tabular data, visuals	Overfits without depth limits
Random forest	Many decorrelated trees average out	Robust default, noisy features	Large memory, weak on smooth trends
Gradient boosting	Weak trees added sequentially reduce error	Best accuracy on tabular data	Sensitive to learning rate, can overfit with too many rounds
SVM (RBF kernel)	Boundary is smooth in a kernel space	Medium-sized, high-dimensional, clean data	Scaling mandatory, slow beyond ~100k rows, no native probabilities

from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC

knn = KNeighborsClassifier(n_neighbors=15, weights="distance")
svc = SVC(C=1.0, gamma="scale", probability=False)

# both are distance-based: scale first, and tune k or C on the validation folds
knn.fit(X_train_scaled, y_train).score(X_test_scaled, y_test)
svc.fit(X_train_scaled, y_train).score(X_test_scaled, y_test)

Trees and ensembles in practice

from sklearn.ensemble import (HistGradientBoostingClassifier,
                              RandomForestClassifier)

forest = RandomForestClassifier(
    n_estimators=500, min_samples_leaf=2, max_features="sqrt",
    n_jobs=-1, random_state=42)

boost = HistGradientBoostingClassifier(
    learning_rate=0.05, max_iter=500, early_stopping=True,
    validation_fraction=0.15, random_state=42)

for name, m in [("forest", forest), ("boost", boost)]:
    m.fit(X_train, y_train)                 # trees do not need scaling
    print(name, m.score(X_test, y_test))

💡

On tabular data a gradient-boosted model is the usual accuracy winner, but only after features are right and the split is honest. Try a regularised linear model first: if it is within a point or two, its coefficients are worth far more than the extra accuracy.

FAQ

Which algorithm should I try first?

A regularised linear model on scaled features, then HistGradientBoostingClassifier as the accuracy baseline. Compare both on the same folds with the same metric before adding anything more exotic.

Why does my SVM take forever?

Kernel SVMs scale roughly quadratically with rows and need the whole training set at prediction time. Above a few tens of thousands of rows, switch to a linear model or a boosted tree ensemble.

Hyperparameter tuning and experiment tracking Unsupervised learning

Last refreshed 2026-09-18.

Supervised algorithms in practice

Start linear

What each family assumes

Trees and ensembles in practice

FAQ

Related