MODEL-PREDICTIVE POLICY LEARNING WITH UNCERTAINTY REGULARIZATION FOR DRIVING IN DENSE TRAFFIC
Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.
The paper addresses the difficulty of covariate shift in model-based reinforcement learning. Here, the distribution over trajectories during is significantly different for the behaviour or data-collecting policy and the target or optimised policy. As a mean to address this, the authors propose to add an uncertainty term to the cost, which is realised by the trace of the covariance of the outputs of a MC dropout forward model. The method is applied to driving in dense traffic, where even single wrong actions can be catastrophic.
I want to stress that the paper was a pleasure to read. It was extraordinarily straightfoward to follow, because the text was well aligned with the necessary equations.
The introduction and related work seem complete to me, with two exceptions:
- Depeweg, S., Hernandez-Lobato, J. M., Doshi-Velez, F., & Udluft, S. 
  (2018, July). Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning. In *International Conference on Machine Learning* (pp. 1192-1201).
- Thomas, Philip S. *Safe reinforcement learning*. Diss. University of Massachusetts Libraries, 2015.
The work by Depeweg et al addresses quite the same question as the authors of this work, but with a broader scope (i.e. not limited to traffic) but very much the same machinery. There are some important theoretical insights in this work and the connection to this submission should be drawn. In particular, the proposed method needs to be either compared to this work or it needs to be clarified why it is not applicable.
The latter appears to be of less significance in this context, but I found robust offline policy evaluation underrepresented in the related work. 
I wonder if there is a way for a neural network to "hack" the uncertainty cost. I suppose that the proposed approach is an approximation to some entropy term, and it would be informative to see how exactly. 
The approach shown by Eq 1 appears to be an adhoc way of estimating whether the uncertainty resulting from an action is due to the data or the model. What happens if this approach is not taken?
The objective function of the forward model is only given in the appendix. I think it needs to be moved to the main text, especially because the sum-of-squares term indicates a homoskedastic Gaussian for a likelihood. This has implications for the uncertainty estimates (see point above).
Overall, the separation of data uncertainty/risk vs model uncertainty is not done. This indicates that heterskedastic environments are candidats where the method can fail, and this limitation needs to be discussed or pointed out.
Further, the authors did not observe a benefit from using a stochastic forward model. Especially, if the prior instead of the approximate posterior is used. My point would be that, depending on the exact grapical model and the way the sampling is done to train the policy, it is actually mathematically *right* to sample from the prior. This is also how it is described in the last equation of section 2. 
## Summary
Overall, I liked the paper and the way it was written. However, there are some shortcomings, such as the comparison to the work by Depeweg et al, which does a very similar thing. Also, justifying the used heuristics as approximations to a principled quantity would help. It appears that the question why and how stochastic forward models should be used requires further investigation.