Most machine learning advice is about training models. Here is what actually matters once a model has to run every day.
Most machine learning advice is about training. Which algorithm, which features, which hyperparameters. That advice is fine as far as it goes, but it covers perhaps a fifth of the work that decides whether a model is still running, and still trusted, a year after launch. This is the other four fifths, written down from projects that shipped and a few that nearly did not.
Start from the decision, not the data
The first question on every project should be: what decision changes because of this model, and who makes it? A demand forecast that nobody in planning looks at is a science project. A fraud score that arrives after the payment cleared is a report.
Writing down the decision does two useful things. It fixes the level of the prediction, whether that is per product per week or per meter per day, so you do not build the wrong model very well. And it gives you a metric in the customer’s units. “Forecast error dropped from 22% to 14%” is good. “Stock-outs fell by a third” is what gets the next project funded.
Beat the baseline before anything else
Every organisation already has a way of predicting the thing you are about to model. Last year plus five percent. The senior planner’s spreadsheet. A rule someone wrote in 2019. Measure it. Then measure your first, simplest model against it, using the same data and the same time split.
If a gradient-boosted model on obvious features cannot beat the baseline in a backtest, the problem is usually the data or the framing, and no amount of architecture will fix it. Finding that out in week two costs a week. Finding it out in month three costs a project.
Time is the feature that breaks everything
The most common silent failure we see is leakage across time. A feature computed with information that was not available at prediction time makes the backtest look wonderful and the production model useless. Customer lifetime value that includes next month’s purchases. A rolling average that peeks at the day being predicted. A label derived from a status field that is updated retroactively.
The defence is boring and non-negotiable: split by time, never randomly, and build features with an explicit “as of” timestamp. If your feature pipeline cannot answer “what did we know on this date”, it is not ready.
Simple models, well maintained, beat clever models, abandoned
We have replaced more than one neural network with a gradient-boosted tree and improved results, not because trees are better, but because the tree was retrained every week and the network had not been touched since the consultant left. A model is a living thing. It needs a retraining schedule, a monitoring dashboard and someone whose job includes looking at that dashboard.
Choose the simplest model that beats the baseline by a comfortable margin, then spend the saved effort on the pipeline around it. That trade almost always pays.
Monitor the inputs, not just the outputs
Accuracy is a lagging indicator. By the time you can measure that last month’s forecasts were wrong, you have already made a month of bad decisions. Input monitoring is the leading indicator: distributions of features, missing-value rates, volume of records, arrival times. When a source system changes a field name or a sensor goes quiet, input monitoring catches it the same day.
A useful minimum: for every feature, track the mean, the missing rate and the share of values outside the training range, and alert when any of them moves more than it has in the last ninety days.
Explanations are a product feature
If a human acts on a prediction, they will ask why. An investigator who cannot see why a meter was flagged will ignore the flag. A planner who cannot see why the forecast jumped will override it. Explanations, even simple ones like the top three factors behind a score, are the difference between a model that is used and a model that is tolerated.
Build them in from the first version. Retrofitting explanations onto a model chosen without them is painful, and the temptation to skip it is strong.
Write the runbook before you leave
The last deliverable on every project should be a document that answers four questions for the client’s team. How does the model get its data, and what does it look like when that breaks? How and when is it retrained? What do the monitoring alerts mean and what should you do about each? And who do you call when none of that helps?
If the answers are “ask the consultant”, the project is not finished.
The short version
Decide first, predict second. Beat the baseline with something simple. Respect time in your features. Monitor inputs. Explain outputs. Leave a runbook. None of this is glamorous, and all of it is what separates the models that are still running from the ones that are still on a slide.