Concept drift essays explain when features stop meaning what they used to. Training–serving skew explains when the live feature pipeline diverges from the training table. Neither answers the operational question teams still get wrong: how do you put a new model into the plant without discovering, three weeks later, that false rejects or missed escapes got worse?
This note is about champion–challenger evaluation—running a candidate beside the current production model until evidence, not optimism, promotes it.
Why offline metrics lie politely
Holdout AUC and confusion matrices are necessary. They are not sufficient. Plant traffic shifts by mix, shift, and supplier lot. Annotation rules drift. A challenger that wins on last quarter’s labeled set can lose on this week’s live stream—especially when the cost of a false reject and an escape are asymmetric. Offline reports rarely price that asymmetry the way the line does.
What a fair online fight requires
- Same features, same clock. Challenger and champion must score identical live feature vectors with aligned timestamps. Joining on approximate keys invents wins.
- Shadow first. Challenger outputs are logged and compared; they do not drive rejects, setpoints, or work orders until a gate says so.
- Paired metrics. Track agreement rate, disagreement clusters by product family, calibrated score gaps, and—when labels arrive late—delayed precision/recall by promotion cohort.
- Human adjudication sample. Disagreements are not only a dashboard; a fixed weekly sample goes to quality for ground truth.
- Promotion and rollback as MoC. Thresholds for promote/rollback are written before the experiment. “Looks better to the data scientist” is not a gate.
What champion–challenger is not
It is not endless A/B on safety-critical hard interlocks. It is not an excuse to skip recipe and model freezes. It is not shadow mode that silently times out into “approve,” and it is not swapping production artifacts because a notebook accuracy cell turned green.
Plants that keep a named champion, a dated challenger run, and a rollback path learn which model actually earns the write path. Plants that only keep the latest file learn later, on scrap and escapes.