Industrial vision projects often stall on “the model is not accurate enough.” Sometimes the network capacity or the lighting is the limit. Often the training labels are inconsistent: the same defect type drawn with different box sizes, borderline marks labeled defect on Monday and acceptable on Tuesday, or two annotators applying different rules to identical images.
Label-audit case studies and lighting-drift notes address related vision failure modes. This article focuses on annotation noise and agreement as a data problem before model choice.
Why noise caps performance
Supervised detectors and classifiers imitate the label distribution they are shown. If the labels encode contradictory human judgments, the optimum the training loss can reach is a compromise—not the plant’s written quality standard. Metrics on a test split drawn from the same noisy process can look optimistic while production disagreements continue.
Editorial practices that reduce noise
Written class definitions with examples. Borderline cases need pictures in the work instruction, not only a one-line class name.
Double annotation on a sample. Measure agreement (even a simple percent agreement or Cohen’s kappa-style summary) before scaling a vendor labeling job. Low agreement means the definition is unfinished.
A frozen gold set. Keep a small, adjudicated image set that does not get “fixed” every time the model fails. Use it to detect definition drift when new annotators join.
Separate hard cases. Ambiguous images belong in a review queue, not forced into train/test with a coin-flip label.
What not to do
Do not “fix” production false rejects by relabeling only the failed images into the accept class without updating the written standard. That teaches the model to match yesterday’s expedience. Do not compare architectures on a dataset whose labels two engineers would not apply the same way.
Vision models are only as coherent as the human rules behind the boxes. Clean the rules and the gold set first; then the choice of backbone has something honest to optimize.