Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation
While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in order to statistically validate performance claims. We implement a simulation framework that can test an entire modern autonomous driving system, including, in particular, systems that employ deep-learning perception and control algorithms. Using adaptive importance-sampling methods to accelerate rare-event probability evaluation, we estimate the probability of an accident under a base distribution governing standard traffic behavior. We demonstrate our framework on a highway scenario, accelerating system evaluation by 2-20 times over naive Monte Carlo sampling methods and 10-300P times (where P is the number of processors) over real-world testing.
This paper introduces a simulation framework -- called a pseudoreality -- that can test an entire autonomous driving system, including deep-learning perception + control algorithms, underlying dynamics models and a photo-realistic rendering engine. They demonstrate full-scale testing using a risk-based framework, evaluating the probability of an accident under some base distribution that stipulates traffic behavior. Adaptive importance sampling methods are used to evaluate risk. The authors demonstrate that system testing can be done by 10-50 times the number of processors used in real-world testing and 1.5-5 times faster than that of naive MC sampling.  The problem domain examined in this paper is one that is extremely relevant, given the introductions of autonomous vehicles onto roads, both in city and highway driving. Due to the rare nature of accidents and uncertainty in real-world environments, testing is computationally prohibitive and so some verification in simulated environments is needed to ensure safety and performance guarantees. A probabilistic "risk-based" framework that simulates rare events only makes sense. The authors try to estimate the probability of a rare event via simulations and generate scenarios that exhibit risky behavior (defined by a threshold parameter).  The key premise of the proposed simulator requires that a model of the world be estimated. This is done by using public traffic data collected by the US DOT (Department of Transportation). Imitation learning is used to learn a generative model for the behavior/policy of environment vehicles, resulting in an ensemble of models that characterize the distribution of human-like driving policies. Specifically, an ensemble of generative adversarial imitation learning models are used to learn the base distribution.  To generate risky scenarios and to estimate the probability of a risky event, the authors use adaptive importance sampling (instead of naive MC sampling) to speed up evaluations. They propose a cross-entropy algorithm that iteratively approximates the optimal importance sampling distribution. Importance sampling + cross-entropy methods choose a distribution over the model parameters in the generative model of human driving behavior. The simulator then constructs a random rollout of behaviors according to the learned policy of a human driver given environmental inputs.  The simulator, then, is the primary contribution of this paper, which is supported by the risk-based framework and cross-entropy search techniques. It can take inputs such as an rendered image and output actuated commands based on an end-to-end highway autopilot network. Testing of the simulator is done using a multi-agent highway scenario where public traffic data is available. The authors show that their simulator accelerates the assessment of rare-event probabilities with respect to real-world testing.   The ideas presented in this paper are interesting in that they attempt to address the problem of how to test and ensure safety and performance of autonomous driving vehicles. Combining adaptive importance sampling + cross-entropy methods in a simulator that attempts to estimate and generate risky events seems like a reasonable approach.  From the paper's exposition, it appears that this is a novel idea and, if so, would be certainly of interest to the machine learning community.  I have read the authors' response and thank them for their comments.