Independent research · autonomous driving
A trajectory predictor made the car brake for parked cars. A pre-registered fix removed 45% of those brakes.
Holding out one of six Argoverse 2 cities costs a learned predictor only about 5% accuracy. In closed loop the bigger problem is elsewhere: it forecasts parked cars pulling away, and the planner brakes for them. Giving stopped agents a constant-velocity forecast cut unnecessary hard brakes by 44.9% on 8,140 scenes no compared model trained on, with collisions inside the registered margin, while a dose-matched sham showed no detectable effect.
At a glance
- 44.9%
- fewer unnecessary hard brakes on 8,140 fresh scenes (CI 37.7 to 51.4%)
- +0.04 pp
- collisions, inside the registered +0.3 pp margin
- 41.0 pp
- larger reduction than a dose-matched sham
- 6.66 ms
- per replan, TensorRT FP16, stopped agents skip inference
The problem
Trajectory predictors are scored open loop, on how close their best of six guesses lands to where each agent actually went. The question that matters for a self-driving car is different: when the predictor is wrong, does the car drive worse?
The study started from a deployment question, whether a predictor knows when it is in a city it never saw, and followed the answer into a planner, because an accuracy drop that never changes a driving decision is not a safety finding.
Method
- Pre-registration
- Each stage's hypotheses, arms, decision rules and kill criteria were committed before that stage was scored, with Bonferroni-adjusted bootstrap intervals.
- Leave one city out
- Six folds, three seeds each: a model trained without a city against an equal-size model trained with it, plus a map-removed positive control.
- Closed loop
- A longitudinal planner replans every second from the predictor's forecasts; the outcome is unnecessary hard braking, with at-fault collisions as the guardrail.
- Dose-matched sham
- The sham gives the same number of random agents constant-velocity forecasts at every replan, so "acting" is separated from "acting on the right agents".
- Fresh-scene replication
- Stage 4 re-ran the fix on 8,140 training scenes that no compared model had trained on, identified by a checksummed pool.
Results
- H1 · City shift
- Miss rate rises 5.3% in an unseen city (CI 3.3 to 7.2%), positive in all six folds: real, but small.
- H2, H3 · Self-knowledge
- Ensemble disagreement still flags the failing predictions in the unseen city (capture fraction 0.333), with no detectable difference from seen cities.
- H8 · The fix
- Constant velocity for stopped agents cut unnecessary hard brakes by 38.4% (CI 34.1 to 42.4%). Development stage: reused validation scenes, and the sham there reached only 43% of the dose; Stage 4 is the confirmation.
- H10 · Replication
- On 8,140 fresh scenes: 44.9% fewer (CI 37.7 to 51.4%), collisions +0.04 pp against a +0.3 pp margin.
- H11 · Specificity
- The fix beat the dose-matched sham by 41.0 pp; the sham alone gave 3.9% fewer brakes with an interval spanning zero.
- Serving
- In TensorRT FP16 on a desktop GPU, letting stopped agents skip inference brings a replan to 6.66 ms and chose the same acceleration as the full path at all 3,000 checked replans (offline replay).
What did not work
Two learned fixes were registered, run, and killed. Retraining on focal plus scored agents cut the stopped-agent miss rate sharply but left braking unchanged (-1.1%, CI -6.3 to +3.6%). A 75/25 mix of focal and stopped-agent samples produced 2.7% more brakes. Both cost focal accuracy.
The likely cause, that Argoverse's focal agents are chosen for being interesting so a stopped focal agent usually moves, fits the data but is not isolated by any intervention in the study.
Limitations
- Harness
- The closed loop is a privileged sensitivity harness, not a driving simulator: other agents replay their logs and do not react, and the planner controls speed only along the logged route. 77 to 84% of at-fault collisions happen after the car passes the end of the human's logged route, so the collision margin mostly measures that extension; scored on the route only, the brake cut is 68% but the registered collision control fails there.
- Scope
- One dataset, six US cities, a small model (1.44M parameters). The stopped-agent failure does not carry over to a stronger released forecaster (QCNet: 2.29% stopped-agent misses vs 9.46% for constant velocity) and disappears at two-second horizons, so it is specific to this model at six seconds.
- Mechanism
- The decision-by-decision mechanism analysis and the planner sweep are exploratory, run after the registered stages.
Reproducibility
The row-level tables for every stage are attached to GitHub releases. Independent audit scripts, which import nothing from the analysis code, re-derive every registered decision from them: point estimates, fresh bootstrap intervals at the registered confidence level, every registered gate, and each verdict.