Safety Critical Labs / Measured Evidence

Verification you can
measure. On public data.

The framework is only as good as the evidence its requirements produce. These are measured results from applying SCL verification methods to a public dataset, NASA C-MAPSS turbofan degradation. Every number is reproducible from the open repository. The methods that failed their own subject are kept in view, because a verification bench that cannot fail is not a verification bench.

[ Subject Under Test ]
Dataset
NASA C-MAPSSFD001 turbofan run-to-failure. Public.
Model
3-class RUL classifierNominal / degraded / critical remaining life
Split
60 / 20 / 20 by unitNo engine appears in two splits
Baseline accuracy
80.3%Held-out test, honest split
AI-1
Leakage caught
A leaky split inflates measured accuracy
+2.72 pp test accuracy when engines are allowed to bleed across train and test, versus a clean unit-level split

Splitting the data by row instead of by engine lets the same asset appear in both training and test. Measured accuracy rises from 80.3% to 83.1%. The gain is not skill. It is memorization the evaluation failed to prevent. A pass threshold set on the leaky number would certify a system that the honest split does not support.

This is the failure mode AI-1 targets: train, validation, and test separation with documented provenance.
AI-4
Monitor qualified
A drift monitor earned through four failed designs
0.58σ / 1.17σ injected bias at first alert and at inhibit, once the monitor was designed against a measured no-drift floor

Naive drift designs fired false alarms on healthy data, population indices past 7.8 where the real signal was noise. Only after declaring regime, per-asset baseline, and window and bin design did the monitor become trustworthy: it alerts at 0.58σ of injected bias and inhibits at 1.17σ. Left unmonitored, the classifier's false-fault rate climbs to 0.70 under full bias.

This is the failure mode AI-4 targets: post-deployment drift monitoring with thresholds meaningful against a sampling floor.
AI-6AI-12
Denial finding
Softmax confidence detects OOD worse than chance
AUROC 0.117 for max-softmax OOD scoring: below 0.5 means the model is most confident exactly when it is most wrong

The popular softmax-confidence detector scores out-of-distribution inputs worse than a coin flip. A Mahalanobis detector recovers gross regime shift perfectly (AUROC 1.0) but catches the subtle near-boundary fan fault only 44% of the time at a 2% false-positive budget. No single OOD method passed every regime. That result is reported as a denial, not smoothed over.

This is the failure mode AI-6 and AI-12 target: a characterized OOD response, not a confidence threshold assumed to work.
AI-8AI-12
Fail → pass
Confidence miscalibrated, then corrected and re-checked
0.160 → 0.030 expected calibration error before and after temperature scaling, against a 0.05 pass criterion

Out of the box the model's stated confidence was off by 0.16 expected calibration error, a clear fail. A single temperature parameter fitted on validation data brought it to 0.030, a pass. The requirement is not that the model is born calibrated. It is that calibration is measured, corrected, and verified before the confidence is trusted.

This is the failure mode AI-8 and AI-12 target: confidence that means what it says, demonstrated by measurement.
[ Why the failures stay in ]

A bench that cannot fail its subject
is not evidence.

Two of the four results above are failures. The softmax detector performs worse than chance. The out-of-the-box calibration does not pass. They are kept because they are the point. A verification method that certifies whatever it is pointed at proves nothing. These are held to the same pass and fail criteria a real assessment would apply, and they are allowed to fail.

[ Reproduce It ]

Open pipeline.
Open data.

The full pipeline, configuration, seeds, and result files are public. Every figure on this page traces to a JSON artifact in the repository, produced under a pre-declared configuration. The framework these methods verify against is openly published under a citable DOI.

Repository github.com/safetycriticallabs/cmapss-verification-demo
Data NASA C-MAPSS FD001. Public, no license restriction on use.
Framework 10.5281/zenodo.19024420 (cite this)

This is a demonstration of SCL verification methods on a public dataset. It is not an issued certification, and it makes no safety claim about any real turbofan engine. It exists to show that the framework's requirements resolve to measurements that can pass, and can fail.