A model that predicts scrap with uncanny offline AUC often fails the first week it scores live. The usual excuses are drift and missing tags. A quieter failure is look-ahead: the training join attached a final-test or disposition label that did not exist at the decision time the features claim to represent. Offline, the label leaks into correlated late sensors. Online, those columns are empty or still zero. Accuracy collapses. Nobody “retrained wrong”—the table lied about when the world was known.
This is not NTP clock skew between historians, not scrap-code taxonomy merges, and not operator-override telemetries. It is event-time honesty in the join.
How the lie is built
A lot row has process tags at t0…tn and a fail flag stamped at final test hours later. A data engineer joins on serial and keeps the fail flag on every early window “for convenience.” A feature that is actually a late metrology result sneaks into the same frame. Cross-validation still looks fine because every fold sees the same contaminated table.
Point-in-time correct training means: for each decision timestamp, only columns that were knowable then. Labels may come from the future; features may not.
Gates before any champion model
Require a point-in-time audit: for a sample of training rows, prove each feature’s source event time is ≤ decision time. Ban joins that broadcast a disposition code onto earlier stations without an explicit “label only” channel that scoring code never reads.
If the online path cannot produce a column, that column must not exist in training—or must be masked identically in offline evaluation. Parity between train and serve is not a slogan; it is a schema contract.
What dashboards should expose
Track train-serve skew: fraction of scored rows with nulls in features that were dense offline, and a weekly sample of decision-time audits. When offline metrics jump after a “data pipeline improvement,” ask whether a future column landed in the frame.
The short rule: the model may learn from yesterday’s failures. It must not be handed tomorrow’s answer dressed as a sensor.