The model lifecycle
From framing a problem to monitoring it in production — and why most AI projects fail before any model is trained.
Frame the problem first
The most common failure is solving the wrong problem precisely. Before any modelling, write down the decision the output will change, who acts on it, and what a wrong answer costs.
| Question | Why it decides everything |
|---|---|
| What decision changes? | No decision means no project |
| What is the cost of a false positive vs false negative? | Sets the metric and the threshold |
| Do we have labels, or must we create them? | Labels are usually the real bottleneck |
| What is the baseline? | A rule or a human; if you cannot beat it, stop |
| How will we notice it degrading? | Decides the monitoring plan |
⚠️
Start from the simplest thing that could work — a rule, a spreadsheet, a keyword search. If a lookup table beats your model, you learned something cheap instead of something expensive.
Data and splits
all data
├── train (fit the parameters) ~70%
├── val (tune settings, early stop) ~15%
└── test (report the final number) ~15% touch it once- Split by entity, not by row: keep all records of one user or one day on the same side, or you leak information and overstate accuracy.
- For time-series, split by time — never randomly.
- Balance checks are part of splitting: look at class frequencies in each split.
- Version your data as carefully as your code; a silent data change invalidates every result.
💡
High accuracy on a skewed dataset is meaningless. With 1% positives, "always predict negative" scores 99% — which is why you track precision, recall or F1 instead.
Evaluation, deployment, monitoring
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score
print(confusion_matrix(y_test, preds))
print(classification_report(y_test, preds))
roc_auc_score(y_test, proba) # threshold-independent ranking quality- Pick one metric tied to the business cost, and report it on the untouched test set.
- Ship behind a shadow or canary path; compare against the existing baseline on live traffic.
- Log inputs, outputs and outcomes so you can measure real performance, not just offline scores.
- Watch for drift: input distributions move, and accuracy decays without any code change.
⚠️
Models leak in subtle ways — an id column that encodes the label, a field filled in only after the outcome is known. If offline scores look too good, hunt for leakage before celebrating.
FAQ
How much data is enough?
Depends on the problem, but the honest answer is: enough that a held-out test set gives a confident comparison against your baseline. For classical ML, hundreds to thousands of rows per class can be plenty; for deep learning, usually far more.
Fine-tune or prompt?
Prompt, add retrieval, or use tools first — they are cheaper and reversible. Fine-tune when you need a consistent format, a specific style, or latency/cost reductions that prompting cannot reach.
Related
Machine learning in one page AI, machine learning, deep learning
Last refreshed 2026-09-18.