Benchmarking Which Safety Nets Catch Self Driving Cars When the World Gets Unfamiliar
Self driving cars can misbehave when they meet scenes they were never trained on, like fog, heavy rain, snow, or images tampered with by an attacker. This paper builds one unified benchmark that tests nine detection methods side by side on both simulated and real roads, and shows which of them most reliably flag those unfamiliar moments.
Self driving cars can misbehave when they meet scenes they were never trained on, like fog, heavy rain, snow, or images tampered with by an attacker. This paper builds one unified benchmark that tests nine detection methods side by side on both simulated and real roads, and shows which of them most reliably flag those unfamiliar moments.
![![Grid of camera views from a self driving car showing normal, foggy, rainy, snowy, and adversarially attacked scenes from two driving datasets]](/news/img/ood-driving-benchmark.JPG)
Why it matters
Autonomous driving systems learn to drive from the images they see during training. On the road they can hit something brand new, a sudden snowstorm, a thick bank of fog, or a picture deliberately altered by an attacker, and the car may not know what it is looking at. When that happens the model can produce a steering angle that is far from normal, and a wrong decision in a moving vehicle is a safety risk. This work studies the detectors whose job is to notice that unfamiliar moment and warn the car before it acts, treating that warning system as part of the software that must be tested.
Existing studies each looked at only a narrow slice, for example just reconstruction based detectors, or just weather, and never put all of them on the same footing. This paper unifies the picture: nine detectors from four families are trained and tested under one identical pipeline, so the comparison is genuinely fair. On top of that it does something most prior work skipped, it adversarially attacks the driving images and checks whether the detector still spots them, which simulates a malicious interference rather than just bad weather. The result is a practical map of which approaches hold up and which fall apart, giving engineers a clear basis for choosing a safety net.
Key findings
- Detectors compared Nine out of distribution detection methods are compared on one shared pipeline.
- Detector families The nine detectors span four families, reconstruction based, density based, distance based, and GAN based.
- Driving datasets Testing covers two driving datasets, one simulated Udacity scene set and one real world Dave dataset.
- Unfamiliar conditions Each detector faces seven out of distribution conditions, three weather shifts and four adversarial attacks.
- OOD images generated The automated pipeline generates 18,228 out of distribution images per weather condition on the real world dataset.
- Best performing family Distance based detectors lead overall with a 0.840 average AUC ROC, the top score among all four families.
- Second best Density based detectors rank second with a 0.782 average AUC ROC and strong robustness.
- Reconstruction weakest Reconstruction based detectors lag at 0.612 AUC ROC, the least reliable family across conditions.
- Real time speed Every detector clears the 30 frames per second real time threshold, so detection is not a bottleneck.
- Fastest detector The fastest detector runs at 1,814 frames per second, ideal for latency critical pre filters.
- Higher is better The study scores detection on AUC ROC, AUC PR, and F1 at five confidence thresholds plus inference speed.
Distance and density based detectors win the safety race
- Distance based detectors deliver the best balance of accuracy and stability across all unfamiliar driving conditions
- Density based detectors are a close second and stay robust under both weather shifts and attacks
- Reconstruction and GAN based detectors fluctuate sharply and are less reliable in the real world
- Detection costs almost nothing, every method clears the real time 30 frames per second bar
What it means for practitioners
When choosing an out of distribution safety net for a perception module, favor distance based or density based detectors. They consistently reach the top statistical groups across weather and adversarial conditions and degrade far more gracefully, which is exactly what a real driving system needs.
Speed is not the obstacle. Every tested detector exceeds 30 frames per second on an A100, and the fastest processes over 1,800 frames per second, so one can be slotted in as a perception pre filter or on an edge device without stalling the pipeline.
Beware simple reconstruction based autoencoders for safety critical role. Although they look good on simulated weather, their metrics collapse on real world data and under attacks, which makes them a poor choice when the cost of a missed detection is a crash.
Related reading
Get the paper & cite it
Official citation: Xiangyu Li, Jingyu Zhang, Jacky Keung, Xiaoxue Ma, Yihan Liao (2026). A Unified Benchmark for Out-of-Distribution Detection for Autonomous Driving Systems.. AST. DOI: 10.1145/3793654.3793758.