RIGOROUS AGENT EVALUATION: AN ADVERSARIAL APPROACH TO UNCOVER CATASTROPHIC FAILURES
This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents. We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation. To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach. Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities. The key difficulty is in identifying these adversarial situations -since failures are rare there is little signal to drive optimization. To solve this we propose a continuation approach that learns failure modes in related but less robust agents. Our approach also allows reuse of data already collected for training the agent. We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving. Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.
PAPER SUMMARY
-------------
The paper proposes a method for evaluating the failure probability of a learned agent, which is important in safety critical domains. 
Using plain Monte Carlo for this evaluation can be too expensive, since discovering a failure probability of epsilon requires on the order of 1/epsilon samples. Therefore the authors propose an adversarial approach, which focuses on scenarios which are difficult for the agent, while still yielding unbiased estimates of failure probabilities. 
The key idea of the proposed approach is to learn a failure probability predictor (FPP). This function attempts to predict at which initial states the system will fail. This function is then used in an importance sampling scheme to sample the regions with higher failure probability more often, which leads to higher statistical efficiency.
Finding the FPP is itself a problem which is just as hard as the original problem of estimating the overall failure probability. However, the FPP can be trained using data from different agents, not just the final agent to be evaluated (for instance the data from agent training, containing typically many failure cases). The approach hinges on the assumption that these agents tend to fail in the same states as the final agent, but with higher probability. 
The paper shows that the proposed method finds failure cases orders of magnitude faster than standard MC in simulated driving as well as a simulated humanoid task. Since the proposed approach uses data acquired during the training of the agent, it has more information at its disposal than standard MC. However, the paper shows that the proposed method is also orders of magnitudes more efficient than a naive approach using the failure cases during training.
REVIEW SUMMARY
--------------
I believe that this paper addresses an important problem in a novel manner (as far as I can tell) and the experiments are quite convincing.
The main negative point is that I believe that the proposed method has some flaws which may actually decrease statistical efficiency in some cases (please see details below).
DETAILED COMMENTS
-----------------
- It seems to me that a weak point of the method is that it may also severly reduce the efficiency compared to a standard MC method. If the function f underestimates the probability of failure at certain x, it would take a very long time to correct itself because these points would hardly ever be evaluated. It seems that the paper heuristically addresses this to some extent using the exponent alpha of the function. However, I think there should be a more in-depth discussion of this issue. An upper-confidence-bound type of algorithm may be a principled way of addressing this problem.
- The proposed method relies on the ability to initialize the system in any desired state. However, on a physical system, where finding failure cases is particularly important, this is usually not possible. It would be interesting if the paper would discuss how the proposed approach would be used on such real systems.
- On page 6, in the first paragraph, the state is called s instead of x as before. Furthermore, the arguments of f are switched.