Deploying and monitoring a model

Batch scoring versus an API, keeping features identical between training and serving, detecting drift, and knowing when to retrain or roll back.

Batch or service

Shape	Latency	Good for	Hard part
Batch scoring	Hours or daily	Churn lists, risk tiers, pricing tables	Scheduling and backfills
Request-time API	Milliseconds	Fraud checks, ranking, recommendations	Latency budget and feature freshness
Streaming	Seconds	Alerting on live events	State, ordering and duplicate events

import joblib
import pandas as pd

model = joblib.load("churn_v3.joblib")

def score_batch(path_in="scores_in.parquet", path_out="scores_out.parquet"):
    rows = pd.read_parquet(path_in)
    rows["churn_prob"] = model.predict_proba(rows)[:, 1]
    rows["scored_at"] = pd.Timestamp.utcnow()
    rows["model_version"] = "churn_v3"
    rows.to_parquet(path_out, index=False)
    return len(rows)

Store the model version with every prediction. Without it you cannot explain an old decision, reproduce a complaint, or compare the behaviour of two releases on live traffic.

Training and serving must agree

# a single feature path shared by training, backfill and the service
def build_features(raw: pd.DataFrame) -> pd.DataFrame:
    out = raw.copy()
    out["amount_log"] = np.log1p(out["amount"])
    out["days_since_signup"] = (out["event_at"] - out["signup_at"]).dt.days
    return out[FEATURE_ORDER]        # column order and names fixed in one place

assert list(build_features(sample).columns) == FEATURE_ORDER

The most common production bug is a feature computed one way in a notebook and another way in the service — a different time window, a different timezone, a different default for missing.
Persist the whole pipeline so encoding and imputation travel with the estimator.
Validate the schema at the boundary: missing columns, unexpected categories, and types that changed upstream.
Log the raw inputs and the feature vector for a sample of requests; that pair is what lets you debug a skew.

Drift, retraining and rollback

import numpy as np

def psi(expected, actual, bins=10):
    """Population stability index between a training column and today's data."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    e = np.histogram(expected, bins=edges)[0] / len(expected)
    a = np.histogram(actual, bins=edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))

# psi under 0.1: stable, 0.1-0.25: watch, above 0.25: investigate
print({c: round(psi(train[c], today[c]), 3) for c in FEATURE_ORDER})

Monitor inputs (drift), outputs (score distribution and positive rate), and outcomes once labels arrive.
A shift in the input distribution is a reason to look, not a reason to retrain. Retrain when the measured metric on recent labelled data falls below the agreed floor.
Keep the previous model artefact and its inputs reproducible so rollback is a config change, not a scramble.
Watch for feedback loops: acting on predictions changes the data the next model will learn from.

⚠️

Define the retraining trigger and the rollback owner before launch. A model that degrades quietly for months while everyone assumes it still works is the most expensive failure mode in production machine learning.

FAQ

How often should I retrain?

On a trigger, not a calendar: retrain when the monitored metric drops below its floor or when input drift crosses the agreed threshold. Fixed schedules are a reasonable fallback when labels arrive slowly.

What should I log?

Model version, feature values, score, and a request identifier; plus the outcome once it is known. Without inputs and outcomes you cannot distinguish drift from an upstream bug.

Interpretability, bias and fairness Data collection, cleaning and feature engineering

Last refreshed 2026-09-18.

Deploying and monitoring a model

Batch or service

Training and serving must agree

Drift, retraining and rollback

FAQ

Related