A quality or soft-sensor model that scores suspiciously high on the offline report and then disappoints on the first live week usually did not “forget” overnight. More often the evaluation never left the training world. Same lots in both splits, features computed with future labels, or a random shuffle across a time series that should have been cut by calendar—any of these inflate AUC while the line stays cold.
Concept drift, training–serving skew, and canary percent gates assume you already measured something real. This note is evaluation-set leakage for plant models—how holdouts get contaminated and how to keep them honest.
Common plant-specific leaks:
- Lot and genealogy duplicates. Sister wafers or reworked serials appear in train and test because the split keyed only on row ID.
- Time leakage. Shuffling minutes from a continuous process puts adjacent samples on both sides; the model memorizes the local regime.
- Label lookahead. A feature built from “final disposition” or a late lab result that would not exist at score time.
- Shared preprocessing fit. Scalers or encoders fit on the full table, then applied before the split—test rows helped define the transform.
- Champion-tuned thresholds on the same holdout used to advertise the score, with no nested or fresh freeze set.
Hygiene that fits a plant MoC
Split by time or by production campaign first, not by random rows. Deduplicate on lot, serial, and recipe family. Freeze the evaluation set with a date and owner; do not quietly add “a few more hard cases” after seeing metrics. Document which features are allowed at inference clock. When a notebook metric jumps after a “small data fix,” assume leakage until proven otherwise.
Cold-start SKU plans and champion–challenger fights need clean scorecards. Leakage makes every gate optimistic. Plants that treat the holdout as a controlled artifact learn what the model earned. Plants that treat it as a folder named test_final_v3 learn on scrap.