Monte Carlo Tree Search (MCTS) has achieved impressive results on a range of discrete environments, such as Go, Mario and Arcade games, but it has not yet fulfilled its true potential in continuous domains. In this work, we introduce TPO, a tree search based policy optimization method for continuous environments. TPO takes a hybrid approach to policy optimization. Building the MCTS tree in a continuous action space and updating the policy gradient using off-policy MCTS trajectories are non-trivial. To overcome these challenges, we propose limiting tree search branching factor by drawing only a few action samples from the policy distribution and defining a new loss function based on the trajectories' mean and standard deviations. Our approach led to some non-intuitive findings. MCTS training generally requires a large number of samples and simulations. However, we observed that bootstrapping tree search with a pre-trained policy allows us to achieve high quality results with a low MCTS branching factor and few simulations. Without the proposed policy bootstrapping, continuous MCTS would require a much larger branching factor and simulation count, rendering it prohibitively expensive. In our experiments, we use PPO as our baseline policy optimization algorithm. TPO significantly improves the policy on nearly all the environments. For example, in complex environments such as Humanoid with a 17 dimensional action space, we achieve a 2.5× improvement over the baseline algorithm.
This paper proposes Tree search Policy Optimization (TPO) algorithm for tasks with continuous action spaces. TPO works after a well trained policy can be obtained (PPO is used in the paper). After a well trained policy is obtained, TPO firstly uses the policy to do Monte Carlo Tree Search (MCTS), with at most 32 actions at each node (state) in the tree (to make MCTS work for continuous action spaces). Then after searching, policy is updated by matching the statistics of MCTS policy (mean and variance), since usually MCTS induces a better policy than the current sampling policy. Experiments on MuJoCo tasks show that TPO has performance improvement over PPO.
1. This paper is basically a special case of dual policy iteration methods (as mentioned), with some heuristics to make MCTS work in continuous action spaces. Unlike dual policy iteration, there is no theoretical justification why doing these heuristics are good/convincible.
2. As a paper focusing on experiments, the results are not enough, in the following senses,
a) There are other existing work of MCTS in continuous action spaces, but they are not mentioned, like
"Monte Carlo Tree Search in Continuous Action Spaces with Execution Uncertainty", Yee et al., 2016.
"Monte Carlo Tree Search for Continuous and Stochastic Sequential Decision Making Problems", Couetoux, 2013.
Without comparing with other baselines, the current experiments are not convincing to claim the proposed TPO method is a better choice over other existing methods.
b) The experiments only show application of TPO on PPO, which does not indicate that TPO can improve performances of other methods, like Soft Actor-Critic (SAC), and "Relative Entropy Regularized Policy Iteration", Abdolmaleki et al, 2018 (RERPI), which are known as better choices for Mujoco tasks. Actually, I was wondering how can TPO work with other true off-policy algorithms like the mentioned SAC and RERPI.
c) The learning results/curves seem to be not efficient. The maximum timestep is 4M. What is timestep? Is it the number of iterations? Does that mean the environment steps are totally 4M *32 = 128 M (since every step there are 32 envs in parallel). Please clarify this, and also show how many environment steps (totally how many number of actions have been taken) are there for PPO here. If their environment steps are not the same, then this comparison is not fair (TPO actually uses more actions). And if 128 M environments are used, then the learning is quite inefficient, for example, the final score for Ant of TPO is about 5000, while SAC achieves 6000 after 3 M environment steps.
3. The MCTS policy update stage (second stage) uses mean and variance matching to update policy Eq. (8). I did not see why this objective is good. There is no intuition or comparison to support it. For the Gaussian policy here, KL divergence between MCTS policy and the current policy induces another different matching of mean and variance. How is this objective compared with KL? Please use experiments to justify.
4. TPO works under restricted requirements (require well performed policies), which makes the claimed contribution of "making MCTS work for continuous action spaces" weak (nearly not hold). Actually, the experiments show that if rho = 0.1, there is no good enough policy, then the proposed method does not work at all. This means the contribution is "MCTS works for continuous action spaces in special cases, that a good policy can be used for sampling". The 32 branch factor is enough also because of this reason. The MCTS in discrete action space is guaranteed to converge (UCT algorithm). However, here there is no evidence that this proposed method will achieve similar results if it starts from initialization rather than well trained policies. From this perspective, I consider this method not really a method that "makes MCTS work for continuous action spaces."
5. What is the replay buffer size for MCTS? Are the trajectories in MCTS buffer going to be used again (or thrown away after calculate the mean and variance of MCTS policy)? I suppose the simulations will be just used for once and the MCTS tree will be thrown away (next iteration a new tree will be constructed). If this is the case, then the proposed method is not really a true "off-policy" method as claimed. It does not have the same sample efficiency as other off-policy methods (like SAC, DQN, DDPG), and it does not need to face the same difficulty like importance ratio corrections. Therefore, it is more a policy update step within dual policy iteration framework, rather than true off-policy learning (with replay buffer storing trajectories, and those trajectories will be reused for multiple times.)
6. TPO requires knowledge of maximum timestep, meaning it is not an any-time algorithm (unlike most existing algorithms).
Overall, this paper has no contributions on theory. And I found the experiments cannot show: 1) the proposed TPO is better than other baselines; 2) TPO can work with other methods, especially off-policy algorithms; 3) the learning efficiency, proposed objective/algorithm are questionable; 4) the proposed method is not really a method that makes MCTS work for continuous action spaces, except with restricted requirements.
=====Update=====
I have read the rebuttal and I keep my rating, since there is no revision to see any improvements.