Support vector machines and kernels

SVC and SVR, the kernel trick, tuning C and gamma, why scaling is mandatory, and when an SVM is the right call at all.

SVC and scaling

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC, LinearSVC

pipe = Pipeline([
    ("scale", StandardScaler()),          # not optional for an SVM
    ("svc", SVC(kernel="rbf", C=1.0, gamma="scale")),
])

pipe.fit(X_train, y_train)
print(pipe.score(X_test, y_test))
print(pipe.named_steps["svc"].n_support_)   # support vectors per class

An SVM finds the boundary with the widest margin between classes. Distance is central to that objective, so a feature measured in thousands dominates one measured in fractions — scaling is part of using the model, not a preprocessing nicety.

Kernels, C and gamma

Kernel	Boundary	Use when
`linear`	A straight hyperplane	Many features, text, data already separable
`rbf`	Smooth, non-linear	The default for dense numeric features
`poly`	Polynomial surfaces	Feature interactions are known to matter
`sigmoid`	Hyperbolic tangent	Rarely better than RBF; mostly historical
`LinearSVC` / `LinearSVR`	Linear, optimised	Large sparse data where kernel SVC is too slow

C controls the cost of violating the margin: large C fits the training data tightly and overfits; small C widens the margin and underfits.
gamma sets the reach of a single training point: large gamma creates tight islands around each point, small gamma makes an almost linear boundary.
gamma="scale" (the default) divides by the feature variance, which is sensible for most dense data; search C first, then gamma.
Only the support vectors determine the boundary, which is why the fitted model stays compact even on large training sets.

search = GridSearchCV(
    pipe,
    {"svc__C": [0.1, 1, 10, 100],
     "svc__gamma": ["scale", 0.01, 0.1, 1.0]},
    scoring="roc_auc", cv=5, n_jobs=-1, refit=True).fit(X_train, y_train)

print(search.best_params_, round(search.best_score_, 4))

When not to use an SVM

Training is roughly quadratic in rows, so beyond a few tens of thousands of examples it becomes impractical.
Prediction requires the support vectors, so latency and memory grow with the retained set.
probability=True builds an internal cross-validated calibration and is much slower — enable it only when you truly need probabilities.
For a large, sparse text problem, LinearSVC or a linear model with a regulariser is usually faster and just as accurate.
The decision function is not interpretable: there are no coefficients to read, so plan for permutation importance if explanation matters.

💡

Use an SVM when the dataset is medium-sized, dense, cleanly scaled, and the boundary is likely non-linear. For tabular data of unknown shape, a gradient-boosted model will typically reach the same accuracy with less tuning and better explanations.

FAQ

What does gamma=scale actually do?

It sets gamma = 1 / (n_features * X.var()). The default adapts to your data, while gamma='auto' uses only the feature count and is rarely better.

How do I get probabilities from an SVC?

Construct it with probability=True, which runs an internal five-fold calibration during fit and adds several hundred milliseconds to the training. Alternatively obtain a decision score from decision_function and calibrate it yourself later.

Linear models Tree-based models

Last refreshed 2026-09-18.

Support vector machines and kernels

SVC and scaling

Kernels, C and gamma

When not to use an SVM

FAQ

Related