Bayesian Policy Optimization for Model Uncertainty
Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world. We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution. Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function. To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state. Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.
Evaluation:
This is a solid paper: The idea is clear, it is well communicated and put into context of the existing literature, and the results are promising. The experiments are well chosen and illustrate the method well. The connection between the chosen setting (BAMDPs) to POMDPs is explained well and explored in the empirical evaluation as well. I think that the methods section could go into a bit more detail, and the underlying assumptions that the authors make could be discussed more critically.
Summary:
This paper looks at Bayes-Adaptive MDPs (BAMDPs) in which the latent parameter space is either
- a discrete finite set or
- a bounded continuous set that can be approximated via discretization.
Consequently, the authors choose to represent the belief as a categorical distribution, which can be represented by a vector of weights.
They further assume that the environment model is known. Hence, the posterior belief can be computed exactly.
If I understand correctly, the main contribution is that the authors represent the policy as a neural network and train it using a policy gradient algorithm.
This is a good first step towards scalable Bayesian policy optimisation.
Main Feedback:
- In the Introduction, first paragraph, you say one of the aspects of real-world robotics is that there's "(1) an underlying dynamical system with unknown latent parameters". I would argue that the dynamic system itself is typically also unknown, including how it is parametrized by these latent parameters. I think it is important to point this out more explicitly in the introduction (it is mentioned in sec 2 and 5, but maybe it's worth mentioning it in 4 again as well): for the problems that you look at, you assume that the form of the transition function is known (just not its parameters phi). 
- In the main methods section (4), it would be nice to see some more detail about the Bayes filter. Can you write out the distribution over the latent parameters, and write out how the filtering is done? Explain how to compute the normalising constant (and mention explicitly why this is possible for your set-up, and why it would be infeasible if the latent space cannot be discretized). How exactly is the posterior distribution represented and fed to the policy? Seeing this done explicitly in Section 4 (even if it repeats some things that are explained in 2) would help someone that is interested in (re-)implementing the proposed method.
- I would like to see a more critical discussion in Section 7 about the assumptions that the authors make: that the environment models are known, and that the latent space can be discretized. How realistic are those assumptions (and in which kind of real-world problems can we make them), and what are ways forward to drop these assumptions?
Other Comments:
- Introduction: Using an encoder for the state/belief is an implementation choice, and (as I see it) not part of the main contribution. I would focus on explaining the intuition behind BPO in the introduction, and only mention the architecture choice as a side note.
- Related Work: The authors might be interested in the recent work of Igl et al. (ICML 2018, "Deep Variational RL for POMDPs"), who approximate the belief in a POMDP using variational inference and a particle filter.
Significance for ICLR:
- In the light-dark experiment, the authors visualise the belief that the agent has at every time step. It would have been nice to see an analysis of how exactly the belief looks also for maybe 1-2 other experiments, and how (when) the agent makes a decision based on this. This could replace Table 2 (which I guess should be called Figure 2?), which I did not find very insightful.