Linear models

Linear and logistic regression, Ridge, Lasso, ElasticNet, how regularisation strength behaves, and how to read coefficients without being misled.

Regression and classification

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

reg = Pipeline([("scale", StandardScaler()),
                ("model", LinearRegression())]).fit(X_train, y_train)

clf = Pipeline([("scale", StandardScaler()),
                ("model", LogisticRegression(max_iter=1000))]).fit(X_train, y_train_cls)

print(clf.predict_proba(X_test)[:, 1][:5])   # positive-class probability
print(clf.named_steps["model"].coef_.shape)  # (1, n_features) binary, (n_classes, n) multiclass
print(clf.named_steps["model"].intercept_)

Logistic regression fits a linear model to the log-odds and predicts a class; the probabilities are a monotone transform of that score. max_iter is raised because the default of 100 often stops before convergence on scaled-but-correlated data.

Regularisation

Estimator	Penalty	Effect
`Ridge`	Squared coefficients (L2)	Shrinks all coefficients; keeps every feature
`Lasso`	Absolute coefficients (L1)	Drives some coefficients to exactly zero — feature selection
`ElasticNet`	L1 plus L2	Sparse like Lasso, stable when features are correlated
`LogisticRegression(penalty=...)`	L1, L2, elasticnet	Same idea for classification
`SGDRegressor` / `SGDClassifier`	Configurable	Large data where full-batch solvers are too slow

from sklearn.linear_model import ElasticNet, Lasso, Ridge

# alpha is the strength: bigger alpha = more shrinkage
for alpha in [0.001, 0.1, 1.0, 10.0]:
    m = Ridge(alpha=alpha).fit(X_train_scaled, y_train)
    print(alpha, round(m.score(X_test_scaled, y_test), 4))

lasso = Lasso(alpha=0.01, max_iter=5000).fit(X_train_scaled, y_train)
print("features kept", (lasso.coef_ != 0).sum(), "of", len(lasso.coef_))

enet = ElasticNet(alpha=0.01, l1_ratio=0.5, max_iter=5000)

alpha (Ridge) and C (logistic, SVM) run in opposite directions: larger alpha means stronger penalty, larger C means weaker penalty.
Always scale before regularising. The penalty is applied to coefficient size, so an unscaled feature with a tiny unit range is penalised far more than an equally useful one measured in thousands.
Lasso is unstable when features are strongly correlated: it picks one and drops its twins, and which one it picks can change with the seed.
Fit the scaler on training folds only — a Pipeline does this for you inside cross-validation.

Reading coefficients

import numpy as np
import pandas as pd

model = clf.named_steps["model"]
pd.DataFrame({
    "feature": feature_names,
    "coef": model.coef_[0],
    "odds_ratio": np.exp(model.coef_[0]),
}).sort_values("coef", key=np.abs, ascending=False).head(10)

For a scaled feature, the coefficient is the change in log-odds per one standard deviation. Exponentiating gives an odds ratio: 1.5 means a standard-deviation increase multiplies the odds by 1.5, holding the other features fixed.

⚠️

Coefficients describe the model, not causality, and they are only comparable when features share a scale. A large coefficient on an unscaled column is often just a unit artefact, and correlated features split their effect unpredictably between them.

FAQ

Why does my logistic regression warn about convergence?

Usually unscaled features, strongly collinear columns, or max_iter set too low. Scale inside a pipeline, raise max_iter to 1000 or more, and check whether a constant column is present.

Ridge or Lasso?

Ridge when you expect many small effects and want stability, Lasso when you believe only a few features matter and want a sparse model, ElasticNet when features are correlated and you want some sparsity without the instability.

Support vector machines and kernels Splitting data and cross-validation strategies

Last refreshed 2026-09-18.

Linear models

Regression and classification

Regularisation

Reading coefficients

FAQ

Related