Continual-retrain holdouts warn against overnight forgetting. Hard-negative mining teaches restraint. Uncertain queues stop forced coin flips. The operational glue many plants skip is simpler: a versioned gold set—adjudicated accepts, rejects, and hard cases—that is re-presented on every labeling batch and every promotion candidate.
Without reseeding, label vendors drift, taxonomies quietly widen, and a “better” model ships that fails last quarter’s escapes. Without a gate that blocks promotion when gold recall or precision moves the wrong way, dashboards celebrate AUC while the line rediscovers old scrap modes.
Keep the gold set small enough to re-label when the SOP revises, large enough to cover each defect family and the hard negatives that define the boundary. Measure inter-annotator agreement on gold periodically; disagreement is a guide problem, not a model problem. Score every adapter and every vendor drop against the sealed set before the line sees weights. When taxonomy changes, either relabel gold or retire cards that no longer match the clause—silent reuse under a new definition is how “we still have a gold set” becomes false assurance.
If gold never fails a candidate, the gate is rubber. If gold never appears in a batch, the set is dead. Keep it alive, or stop calling the process governed.