A defect model can look strong on a holdout while the floor argues about every borderline scratch. Often the training set never had a stable definition. Two annotators marking the same image cohort disagree, the “gold” label is whoever clicked last, and the network learns that coin-flip as truth.
Vision annotation label noise covers corrupted tags and easy-case bias in active learning. Evaluation-set leakage covers contaminated holdouts. This note is inter-annotator agreement as a gate for plant vision labels—before champion metrics get airtime.
Practical pattern: hold out a fixed overlap set. At least two qualified annotators label it independently under the written defect dictionary. Compute agreement (percent agreement plus a chance-corrected measure such as Cohen’s κ for categorical defect codes—use the metric your quality system standardizes). Investigate classes with low agreement before you buy more images. If κ is poor, stop model training and fix the definition, lighting SOP, or training of annotators.
What agreement is not
It is not a substitute for gauge R&R on the camera system. It is not automatic proof that the dictionary matches customer escapes. It is not permission to average conflicting labels into a soft target and call it done without a disposition rule.
Ship models against datasets that survived an agreement gate dated and owned like any other MoC artifact. Skip the gate and the model will confidently automate the argument the inspectors were already having—only faster, and on every shift.