Training–serving skew is when train and live pipelines compute different features. Score calibration drift is when probabilities no longer match outcomes after a base-rate change. Concept drift (and the related covariate shift of input distributions) is when the relationship—or the inputs themselves—move while the model artifact stays frozen.
Label noise and prediction-interval briefs own other AI failure modes. This article owns detecting that the live world left the training distribution.
What moves in plants
A resin lot with a wider melt index. A new coating thickness. A thermocouple that was recalibrated five degrees. A seasonal humidity swing on a drying oven. None of these need a malicious actor. They change the joint distribution of features and quality. A model trained last quarter keeps outputting confident scores on a shifted manifold.
Monitors that earn their keep
Track population statistics on key features (means, quantiles, missingness) against a frozen training reference window. Track residual or error rates on a small, regularly labeled hold sample—not only on auto-accepted production. Alert on sustained divergence before you “retrain because scrap spiked.” Separate sensor faults (one tag broken) from true process shift (many related features moved together).
What not to do
Do not silence drift alerts by widening thresholds until nothing fires. Do not retrain on the last week of polluted labels without a MoC that states why the old concept is retired. Do not confuse a one-day outlier campaign with a new steady state.
If features still arrive and the model still scores, the plant can look automated while the advice quietly describes yesterday. Watch the inputs and the residuals; then decide whether to freeze, recalibrate, or retrain.