Plant ML failures are often blamed on “drift” or “the model went stale.” Sometimes the distribution did shift. Sometimes training and serving never computed the same thing.
Historian alias collisions and NTP clock skew poison labels and time. Label noise caps vision ceilings. This article is about training–serving skew: feature logic that diverges between the notebook that built the model and the service that scores live tags.
Two pipelines, one name
Training pulls a historical extract: joins, resampling, unit conversion, clipping, imputation, and lag features assembled under one script version. Serving reads live tags through a different path—PLC alias, edge aggregator, or a “temporary” SQL view—and applies transforms that someone shortened to meet latency. The model file is identical. The feature vector is not.
Skew that looks like mild drift
A lag that was 5 minutes in training and 1 sample in serving. A temperature that was °C in the training warehouse and °F on the live tag with no conversion. A one-hot for grade that drops an unknown level to zeros in training but maps to a default in production. Offline AUC stays pretty; online advice systematically misses one product family.
Controls that reduce the lie
Version the feature code the same way you version the model artifact. Prefer one shared library or definition for train and serve. Shadow-score: compute training-style features on a recent window and compare to live feature logs before promoting. Put feature-version and model-version on every scored event so post-mortems can see mismatch instead of arguing about “AI quality.”
Do not fix skew by retraining on the broken live path without documenting that the live path is now the standard—you will only teach the model to expect the bug.
If training and serving disagree on the inputs, no amount of hyperparameter search repairs the recommendation. Align the features first; then argue about the algorithm.