On simulator-rendered input, a production driving model goes blind without saying so

supercombo is the end-to-end network that drives openpilot, an L2 driver-assistance system running on consumer hardware on public roads. It exports predictive-uncertainty heads, which downstream code and human intuition both treat as a competence signal.

The safety question is narrow and answerable: presented with input drawn from outside its training distribution, does the model fail conspicuously or silently?

Most OOD-detection work trains a fresh model with a detector attached, on an academic benchmark. That tells you little about a checkpoint already deployed on real roads, where you cannot retrain, cannot add a loss term, and can only observe what the shipped artifact chooses to expose.

The other common failure is uncontrolled teardown: instrument a model, observe strange behavior, and attribute it to the model when it was really the harness. Any claim here is worthless without parity first.

Establish parity before pathology. I reimplemented openpilot v0.9.7 inference until it agreed with comma's own reference output on real footage for 100% of 1,159 frames within ±0.5 m/s² (median absolute deviation 0.04 m/s²). Only after that control was locked does a collapse become attributable to the model rather than to my code.

The second insight: instrument the internal recurrent state, not just the outputs. The failure is legible there even when the model's own interface stays flat.

Parity control

Frame-exact reproduction against comma's reference outputs on real driving footage before any shifted input was run.

Recurrent state threading

The model is temporal: its state buffers are rolled correctly between frames, since a mis-threaded state fabricates collapse on its own.

Full instrumentation

Every output head, every exported uncertainty, and the 512-D recurrent feature vector recorded per frame.

Alpha sweep

A continuous real-to-CARLA blend rather than a binary comparison, which is what exposed the onset as a hard discontinuity.

Threshold-free evaluation

AUROC, bootstrap confidence intervals, and leave-one-corpus-out across four real corpora, so the detector is never scored on the corpus that tuned it.

E1 · Output collapse

8 of 10 tracked output readouts fall below 1% of their real-footage temporal activity on simulated input.

E2 · Latent collapse

The 512-D recurrent feature contracts to 0.00001× the real spread: 219 distinct frames map onto effectively one point.

E3 · Useless uncertainty

Outputs shed about 99.5% of their activity, yet predicted uncertainty rises only 1.2 to 1.8×, and 0 of 219 simulator frames push its exported uncertainty channels past their real-driving range.

E4 · Discontinuous onset

Under the blend, activity first inflates to 6.3× baseline, then collapses across a hard discontinuity at about 78% CARLA. Uncertainty never responds.

E5 · Encoder alive

Per-stage activity holds at or above baseline throughout, placing the earliest contraction downstream of the encoder, in the recurrent and policy stack.

E6 · Recoverable internally

A percentile threshold on the rolling spread of the model's own recurrent vector fires well ahead of the discontinuity, at a 2.41% mean held-out FPR (95% CI [0%, 5.17%], worst fold 6.90%). The repository labels this deployment-unsupported: a fleet-scale false-positive rate is unproven.

E7 sets the bound. Across 15 ImageNet-C corruptions at 5 severities, the E6 detector largely fails: mean AUROC 0.52 to 0.74, near chance at the low end. It is a collapse detector, and photometric corruption is not collapse. Feature-space baselines such as Mahalanobis recover much of what it misses.

That bounds the contribution. The finding is that this specific failure mode is detectable internally and invisible externally, not that I have built a general-purpose OOD monitor.

A downstream safety monitor consuming this model's published interface would see nothing wrong at the exact moment the model had stopped perceiving. The signal needed to catch it exists inside the network and is simply never exposed, which is an architectural decision rather than a fundamental limit.

Every public claim and its boundary is registered in the repository's evidence register. Figures regenerate from cached artifacts without a GPU, and the environment is pinned.

Reproduce from cache

Environment

pip install -r requirements.lock.txt, then make figures regenerates every figure from cached runs.

E1 output collapse

8/10 readouts below 1% of real-footage temporal activity.

E3 uncertainty

0 of 219 simulator frames exceed the real-driving p95 on the exported uncertainty channels.

E6 held-out FPR

2.41% leave-one-corpus-out, 95% CI [0.00, 5.17].

E7 coverage

AUROC 0.52 to 0.74 across ImageNet-C, which bounds the detector.

The reusable component is the method, not the checkpoint: parity control, state threading, alpha sweep, then threshold-free evaluation with leave-one-corpus-out. It is now the standard teardown procedure I apply to any shipped model.