System: Inline vision QC · cosmetic + critical defect class(익명)
Claim: “99% accurate”
Shipment incident: Critical defect escaped
Format: Confusion-matrix autopsy. Vendor accuracy slide 아님.
Matrix(validation week, counts)
Pred OK Pred NG
Actual OK 9800 120
Actual NG (cos) 40 600
Actual NG (crit) 14 26
Accuracy = (9800+600+26) / total ≈ 99.1%.
Critical escape rate = 14 / (14+26) = critical 35% missed.
Accuracy congratulated itself. Customer did not.
Dangerous cell은 accuracy가 advertise하는 cell rarely.
What the threshold did
Operating point false NG minimize(scrap cost visible).
Critical false OK KPI board off.
Threshold theater wrong cell optimize.
Class imbalance honesty
OK dominates. Model sleep해도 accurate look.
Required metrics going forward:
| Metric | Gate | | --- | --- | | Critical recall | ≥ agreed % | | Cosmetic precision | ≥ agreed % | | Escape budget / week | hard cap | | Accuracy | not a gate |
Data sins found
- Critical example rare in training; augmented cosmetic padded set.
- Lighting change after training; image stats drift monitor 없음.
- Operator NG→OK override label feedback 없음.
Override without label feedback next escape train.
Disposition
- Critical class model quarantine; recall gate met까지 cosmetic-only
- Dual-gate: vision NG or critical feature sampling plan
- Escape budget owner named weekly matrix review
Adjacent fences
Soft-sensor model card는 process CV proxy. Unsupervised drift는 unlabeled hypothesis. Write-path red-team은 actuation. Synthetic-data vision essay는 training trick. This autopsy why accuracy lied. Critical recall bleed하는데 99% celebrate 마라.
Close
Escape cell circled matrix publish—or customer defect never met confidence interval ship 계속.