HowToCodePage

Hyperparameter tuning and experiment tracking

Grid, random and Bayesian search, early stopping as a cheap budget saver, logging runs and seeds, and reporting uncertainty on the result.

StrategyCostBest for
Grid searchGrows multiplicativelyTwo or three parameters with a clear range
Random searchFixed budget, any sizeWide spaces; most parameters barely matter
Bayesian / TPESequential, needs a libraryExpensive fits where each trial is costly
Manual + domain knowledgeAlmost freeChoosing the ranges and the metric in the first place
from scipy.stats import loguniform, randint
from sklearn.model_selection import RandomizedSearchCV

search = RandomizedSearchCV(
    estimator=pipe,
    param_distributions={
        "clf__C": loguniform(1e-3, 1e2),          # sample on a log scale
        "clf__penalty": ["l2"],
        "clf__class_weight": [None, "balanced"],
    },
    n_iter=40, scoring="average_precision",
    cv=5, n_jobs=-1, random_state=42, refit=True)

search.fit(X_train, y_train)
print(search.best_params_, round(search.best_score_, 4))

Tune two or three parameters, not everything. Each extra dimension multiplies the fits, increases the chance of overfitting the validation folds, and makes the winner harder to explain.

Spend the budget wisely

from sklearn.ensemble import HistGradientBoostingClassifier

boost = HistGradientBoostingClassifier(
    learning_rate=0.05,
    max_iter=1000,
    early_stopping=True,          # stop when validation loss stops improving
    validation_fraction=0.15,
    n_iter_no_change=25,
    random_state=42).fit(X_train, y_train)

print(boost.n_iter_)              # rounds actually used
  • Early stopping inside the estimator is usually a bigger win than a fine-grained search over the number of rounds.
  • Start coarse: widen the learning rate and depth ranges before narrowing anything.
  • Tune on a subsample first when each fit takes minutes; confirm the shortlist on the full folds.
  • Keep the evaluation protocol fixed while tuning. Changing the folds mid-search makes the trials incomparable.

Track runs and seeds

import json, time
from pathlib import Path

def log_run(params, scores, metric, notes, path="runs.jsonl"):
    row = {
        "ts": time.time(),
        "params": params,
        "mean": float(scores.mean()),
        "std": float(scores.std()),
        "folds": [round(float(s), 4) for s in scores],
        "scoring": metric,
        "n_train": int(len(X_train)),
        "seed": params.get("random_state"),
        "notes": notes,
    }
    with Path(path).open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(row) + chr(10))

log_run(search.best_params_, scores, "average_precision", "log-uniform C")
💡
Log the metric together with the fold scores, the data snapshot and the code revision. A result without its environment is a rumour: you cannot reproduce it, compare it fairly, or explain it to a reviewer six months later.

FAQ

How many trials are enough?
Enough to see the metric plateau. In practice 30 to 60 random trials over two or three parameters, or a few dozen Bayesian trials, captures most of the available gain on tabular data. Beyond that you are fitting the validation folds.
Should I fix the random seed?
Yes while developing, so a changed score means a changed model. Then re-run the winner with several seeds and report the spread — that tells you how much of the improvement is real.

Splitting data correctly Classification metrics in depth

Last refreshed 2026-09-18.