Bayesian Policy Optimization for Model Uncertainty
Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world. We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution. Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function. To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state. Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.
 In this paper, the author proposed to utilize a novel policy structure and recent batch policy optimization methods such as PPO or TRPO to solve Bayes-Adaptive MDP (BAMDP) and Partial Observable MDP(POMDP) problems. The author verified the proposed method on discrete and continuous POMDP and BAMDP benchmarks compared with other baseline methods. The main part of the paper is trying to explain Bayesian RL and the relationship between BAMDP and POMDP, and several related work. There is only a half page that explains the main idea of the proposed method, and it seems that the author combines several existing techniques and utilize deep learning to solve BAMDP and POMDP problems. The detail of the experiment is not clarified explicitly, such as the structure and size of the policy, training details of the BPO, and detail parameters changed to formulate BAMDP for Mujoco environments. The paper strikes me as a valuable contribution once the detail of the experiments are addressed, but personally I am not sure that whether the novelty of this paper is enough for the main conference track.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Bayesian Policy Optimization for Model Uncertainty
Addressing uncertainty is critical for autonomous systems to robustly adapt to the real world. We formulate the problem of model uncertainty as a continuous Bayes-Adaptive Markov Decision Process (BAMDP), where an agent maintains a posterior distribution over latent model parameters given a history of observations and maximizes its expected long-term reward with respect to this belief distribution. Our algorithm, Bayesian Policy Optimization, builds on recent policy optimization algorithms to learn a universal policy that navigates the exploration-exploitation trade-off to maximize the Bayesian value function. To address challenges from discretizing the continuous latent parameter space, we propose a new policy network architecture that encodes the belief distribution independently from the observable state. Our method significantly outperforms algorithms that address model uncertainty without explicitly reasoning about belief distributions and is competitive with state-of-the-art Partially Observable Markov Decision Process solvers.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 In this paper, the author proposed to utilize a novel policy structure and recent batch policy optimization methods such as PPO or TRPO to solve Bayes-Adaptive MDP (BAMDP) and Partial Observable MDP(POMDP) problems. The author verified the proposed method on discrete and continuous POMDP and BAMDP benchmarks compared with other baseline methods. The main part of the paper is trying to explain Bayesian RL and the relationship between BAMDP and POMDP, and several related work. There is only a half page that explains the main idea of the proposed method, and it seems that the author combines several existing techniques and utilize deep learning to solve BAMDP and POMDP problems. The detail of the experiment is not clarified explicitly, such as the structure and size of the policy, training details of the BPO, and detail parameters changed to formulate BAMDP for Mujoco environments. The paper strikes me as a valuable contribution once the detail of the experiments are addressed, but personally I am not sure that whether the novelty of this paper is enough for the main conference track.
assistant
 In this paper, the author proposed to utilize a novel policy structure and recent batch policy optimization methods such as PPO or TRPO to solve Bayes-Adaptive MDP (BAMDP) and Partial Observable MDP(POMDP) problems. The author verified the proposed method on discrete and continuous POMDP and BAMDP benchmarks compared with other baseline methods. The main part of the paper is trying to explain Bayesian RL and the relationship between BAMDP and POMDP, and several related work. There is only a half page that explains the main idea of the proposed method, and it seems that the author combines several existing techniques and utilize deep learning to solve BAMDP and POMDP problems. The detail of the experiment is not clarified explicitly, such as the structure and size of the policy, training details of the BPO, and detail parameters changed to formulate BAMDP for Mujoco environments. The paper strikes me as a valuable contribution once the detail of the experiments are addressed, but personally I am not sure that whether the novelty of this paper is enough for the main conference track.