Skip to content

Industrial technology

← All analysis

Sector · AI · 29 Jul 2026

Confusion matrix post-mortem: the vision model that was ‘99% accurate’

False escapes, class imbalance, and threshold theater decide whether vision QC protects the customer—industrial AI evaluation autopsy, not soft-sensor model cards, not unsupervised drift essays, not write-path red-teams.

Confusion matrix post-mortem: the vision model that was ‘99% accurate’

System: Inline vision QC · cosmetic + critical defect classes (anonymized)
Claim: “99% accurate”
Shipment incident: Critical defect escaped
Format: Confusion-matrix autopsy. Not a vendor accuracy slide.

Matrix (validation week, counts)

                 Pred OK    Pred NG
Actual OK          9800        120
Actual NG (cos)      40        600
Actual NG (crit)     14         26

Accuracy = (9800+600+26) / total ≈ 99.1%.
Critical escape rate = 14 / (14+26) = 35% of criticals missed.

Accuracy congratulated itself. The customer did not.

Confusion matrix printed and circled at the false-negative cell

The dangerous cell is rarely the one accuracy advertises.

What the threshold did

Operating point tuned to minimize false NG (scrap cost visible).
Critical false OK stayed off the KPI board.
Threshold theater optimized the wrong cell.

Class imbalance honesty

OK dominates. A model can sleep and look accurate.
Required metrics going forward:

| Metric | Gate | | --- | --- | | Critical recall | ≥ agreed % | | Cosmetic precision | ≥ agreed % | | Escape budget / week | hard cap | | Accuracy | not a gate |

Data sins found

  1. Critical examples rare in training; augmented cosmetics padded the set.
  2. Lighting change after training; no drift monitor on image stats.
  3. Operator overrides of NG→OK not fed back as labels.

Inline vision camera over a moving production line

Overrides without label feedback train the next escape.

Disposition

  • Quarantine model for critical class; cosmetic-only until recall gate met
  • Dual-gate: vision NG or sampling plan for critical features
  • Weekly matrix review with escape budget owner named

Adjacent fences

Soft-sensor model cards own process CV proxies. Unsupervised drift owns unlabeled hypotheses. Write-path red-teams own actuation. Synthetic-data vision essays own training tricks. This autopsy owns why accuracy lied. Do not celebrate 99% while critical recall bleeds.

Close

Publish the matrix with the escape cell circled—or keep shipping confidence intervals that never met the customer’s defect.

More in this sector