Flagship 02 · Independent research · openpilot

Does a production self-driving model know when it is blind?

A parity-controlled teardown of the neural network that drives openpilot on public roads. Under distribution shift its perception collapses completely, and not one signal it exports would tell a downstream monitor that anything is wrong.

Repository Evidence register Demo video

1,159 frames matched to comma's reference within ±0.5 m/s²

8 / 10 output readouts below 1% of real activity

0% shifted frames exceeding nominal uncertainty

2.41% held-out FPR of the internal-state monitor

0.52 worst-case detector AUROC on photometric corruption

01 The problem

supercombo is the end-to-end network that drives openpilot, an L2 driver-assistance system running on consumer hardware on public roads. It exports predictive-uncertainty heads, which downstream code and human intuition both treat as a competence signal.

The safety question is narrow and answerable: presented with input drawn from outside its training distribution, does the model fail conspicuously or silently?

02 Why existing approaches fall short

Most OOD-detection work trains a fresh model with a detector attached, on an academic benchmark. That tells you little about a checkpoint already deployed on real roads, where you cannot retrain, cannot add a loss term, and can only observe what the shipped artifact chooses to expose.

The other common failure is uncontrolled teardown: instrument a model, observe strange behavior, and attribute it to the model when it was really the harness. Any claim here is worthless without parity first.

03 Core insight

Establish parity before pathology. I reimplemented openpilot v0.9.7 inference until it agreed with comma's own reference output on real footage for 100% of 1,159 frames within ±0.5 m/s² (median absolute deviation 0.04 m/s²). Only after that control was locked does a collapse become attributable to the model rather than to my code.

The second insight: instrument the internal recurrent state, not just the outputs. The failure is legible there even when the model's own interface stays flat.

04 Method

Parity controlFrame-exact reproduction against comma's reference outputs on real driving footage before any shifted input was run.

Recurrent state threadingThe model is temporal: its state buffers are rolled correctly between frames, since a mis-threaded state fabricates collapse on its own.

Full instrumentationEvery output head, every exported uncertainty, and the 512-D recurrent feature vector recorded per frame.

Alpha sweepA continuous real-to-CARLA blend rather than a binary comparison, which is what exposed the onset as a hard discontinuity.

Threshold-free evaluationAUROC, bootstrap confidence intervals, and leave-one-corpus-out across four real corpora, so the detector is never scored on the corpus that tuned it.

05 Results

E1 · Output collapse8 of 10 tracked output readouts fall below 1% of their real-footage temporal activity on simulated input.

E2 · Latent collapseThe 512-D recurrent feature contracts to 0.00001× the real spread: 219 distinct frames map onto effectively one point.

E3 · Useless uncertaintyOutputs shed about 99.5% of their activity, yet predicted uncertainty rises only 1.2 to 1.8×, and 0% of shifted frames exceed the model's nominal real-driving range.

E4 · Discontinuous onsetUnder the blend, activity first inflates to 6.3× baseline, then collapses across a hard discontinuity at about 78% CARLA. Uncertainty never responds.

E5 · Encoder alivePer-stage activity holds at or above baseline throughout, placing the earliest contraction downstream of the encoder, in the recurrent and policy stack.

E6 · Recoverable internallyA percentile threshold on the rolling spread of the model's own recurrent vector fires well ahead of the discontinuity, at a 2.41% mean held-out FPR (95% CI [0%, 5.17%]).

06 What did not work

E7 is the result I would have preferred to leave out. Across 15 ImageNet-C corruptions at 5 severities, the E6 detector largely fails: mean AUROC 0.52 to 0.74, near chance at the low end. It is a collapse detector, and photometric corruption is not collapse. Feature-space baselines such as Mahalanobis recover much of what it misses.

That bounds the contribution honestly. The finding is that this specific failure mode is detectable internally and invisible externally, not that I have built a general-purpose OOD monitor.

07 Significance

A downstream safety monitor consuming this model's published interface would see nothing wrong at the exact moment the model had stopped perceiving. The signal needed to catch it exists inside the network and is simply never exposed, which is an architectural decision rather than a fundamental limit.

08 Limitations

ScopeOne model family (openpilot v0.9.7 and v0.9.6), one simulator (CARLA), two vehicle corpora.

Shift typeSimulator rendering, not real-world night or glare. Real degraded footage did not collapse the model; the failure is sim-specific.

Detector coverageBounded, as E7 records. Not validated as a general OOD detector.

PublicationManuscript drafted and banked as a technical report. Not submitted, not peer reviewed.

09 Reproducibility

Every public claim and its boundary is registered in the repository's evidence register. Figures regenerate from cached artifacts without a GPU, and the environment is pinned.

reproduce from cache

$ pip install -r requirements.lock.txt $ make figures # regenerate from cached runs E1 output collapse ..... 8/10 readouts < 1% E3 uncertainty ......... 0/219 exceed real p95 E6 LOCO held-out FPR ... 2.41% CI [0.00, 5.17] E7 ImageNet-C coverage . AUROC 0.52-0.74 (bounded)

10 Next step

The reusable component is the method, not the checkpoint: parity control, state threading, alpha sweep, then threshold-free evaluation with leave-one-corpus-out. It is now the standard teardown procedure I apply to any shipped model, and it is what the reliability-shield work runs on next.