Informed and robust decision making in the face of uncertainty is critical for robots that perform physical tasks alongside people. We formulate this as a Bayesian Reinforcement Learning problem over latent Markov Decision Processes (MDPs). While Bayes-optimality is theoretically the gold standard, existing algorithms do not scale well to continuous state and action spaces. We propose a scalable solution that builds on the following insight: in the absence of uncertainty, each latent MDP is easier to solve. We split the challenge into two simpler components. First, we obtain an ensemble of clairvoyant experts and fuse their advice to compute a baseline policy. Second, we train a Bayesian residual policy to improve upon the ensemble's recommendation and learn to reduce uncertainty. Our algorithm, Bayesian Residual Policy Optimization (BRPO), imports the scalability of policy gradient methods as well as the initialization from prior models. BRPO significantly improves the ensemble of experts and drastically outperforms existing adaptive RL methods.
This paper considers Bayesian Reinforcement Learning problem over latent Markov Decision Processes (MDPs). The authors consider making decisions with experts, where each expert performs well under some latent MDPs. An ensemble of experts is constructed, and then a Bayesian residual policy is learned to balance exploration-exploitation tradeoff. Experiments on Maze and Door show the advantages of residual policy learning over some baselines.
1. The Bayesian Reinforcement Learning problem this work considered is important. However, using experts immediately make the problem much easier. The original Bayesian Reinforcement Learning problem is then reduced to making decision with experts. Under this setting, there are many existing work with respect to exploration-exploitation tradeoff (OFU, Thompson Sampling) with theoretical guarantees. I did not see why using this residual policy learning (although as mentioned residual/boosting is useful under other settings) is reasonable here. There is not theoretical support showing that residual learning enjoys guaranteed performance. The motivation of introducing this heuristic is not clear.
2. The comparisons with UPMLE and BPO seems not convincing. Both BPO and UPMLE do not use experts, and ensemble of experts outperforms them as shown in the experiments. And the ensemble baseline here is kind of weak (why sensing with probability 0.5 at each timestep?) Always 0.5 does not make sense (exploration should decrease as uncertainty reduced). Other exploration methods should be compared, to empirically show the advantages/necessities of residual policy learning.
Overall, I consider the proposed BRPO a simple extension of BPO, with a heuristic of learning ensemble policy to make decisions. BRPO is lack of theoretical support, and it is not clear why residual policy learning here is necessary and what exactly the advantage is over other exploration methods. Comparisons with simple baseline like exploration with constant probability is not enough to justify the proposed method.
=====Update=====
Thanks for the rebuttal. The comparison with PSRL improves the paper. However, I still think this paper needs more improvement as follows.
Theorem 1 looks hasty to me. Batch policy optimization Alg is going to solve n_{sample} MDPs, which are generated from P_0. But Eq. (6) or Theorem 1 does not contain information about P_0, implying that P_0 has no impact, which is questionable (an uniform P_0 that can generate different MDPs and a deterministic P_0 can only generate one MDP should be very different). I suggest the authors do more detailed analysis.
On the other hand, I expected whether this special "residual action" heuristic has any guarantees in RL? Can decomposing action into a_r + a_e provide us a better exploration method (than others like PSRL, OFU...)? Since this is the main idea of this paper as an extension of BPO, I think this point is important. The experiments shows that it can work in some cases, but I do not see an explanation (the "residual learning" paragraph is high level and I do not get an insight from that.).