Bayes-Adaptive Simulation-based Search with Value Function Approximation
Bayes-adaptive planning offers a principled solution to the explorationexploitation trade-off under model uncertainty. It finds the optimal policy in belief space, which explicitly accounts for the expected effect on future rewards of reductions in uncertainty. However, the Bayes-adaptive solution is typically intractable in domains with large or continuous state spaces. We present a tractable method for approximating the Bayes-adaptive solution by combining simulationbased search with a novel value function approximation technique that generalises appropriately over belief space. Our method outperforms prior approaches in both discrete bandit tasks and simple continuous navigation and control tasks.
This paper deals with Bayes-adaptive model-based reinforcement learning. The underlying principle of this type of approach is to maintain a Bayesian posterior over dynamics (conditioned on past experienced transitions) and to seek at each time step for the action optimizing the related augmented MDP (on state-history meta-states and related meta-dynamics), which is generally an intractable problem (for exact solving). This contribution relies on two previous ideas, simulation-based search (with "root sampling", which avoids updating the belief over dynamics during planning) and value function approximation (which requires introducing proper features for handling histories), combining them to provide a new approach. In addition to this general approach, called BAFA, the authors provide an alternative and more general proof for the validity of root sampling and provide some experimental results.Overall, the paper is well written and clear, it proposes a sound approach, based on known ideas but combining them smartly (especially for the history features, which seems to be the newest part/the core contribution). My major comment is that experiences, if convincing, are not detailed enough to be reproducible (for example, some meta-parameters are not provided, nor discussed). There might also be some problems with corollary 1 (is the policy really stationary?).Detailed (mostly minor) comments:* Line 106, "belief" can be understood as a synonym of "history" in this sentence, which is misleading* In the pseudo-code, Simulate(...,d) should be Simulate(...,t), the "+" in the parameters update (Q-value) should be a "-"* Line 103, a weighted squared loss is mentioned, what are the weights? (do not appear in the next equation, nor in the update rule)* Line 153, it is said that it is necessary to show that root sampling is valid to exploit general convergence results of classical RL. That's right, but is this sufficient?* Lines 190-192 (and Corollary 1). I agree with this remark, stating that using a fixed policy implies the same distribution of parameters update, in light of Lemma 1. However, this is not what does the provided algorithm (at least as stated in the pseudo-code): the considered policy is epsilon-greedy respectively to Q_w, but the parameters are updated during planning, so the policy is not stationary. The policy should remain fixed for at least each drawn dynamics (if this is actually the case, the pseudo-code should be more clear).* For the Bernoulli Bandit, has gamma any importance? How does this compare to standard bandit algorithms?* Generally, for the experiments, more details could be provided (learning rates for the function estimation, more details about parameterizations, etc.). Ideally, they should be reproducible.* Fig. 2, K already denotes the kernel of the GP, choose another notation?* In the height map problem, it is rather strange to consider the reward z as a state component. Why is this done?* When the dynamics distribution is modeled as a GP, cannot the value function be modeled as a GP too (not necessarily straightforward, due the bilinear representation, but maybe an interesting perspective)?* Some uppercases are missing in the bib (eg, pomdps in ref 20)