Surprising Negative Results for Generative Adversarial Tree Search
While many recent advances in deep reinforcement learning rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment dynamics. One prospect is to pursue hybrid approaches, as in AlphaGo, which combines Monte-Carlo Tree Search (MCTS)—a model-based method—with deep-Q networks (DQNs)—a model-free method. MCTS requires generating rollouts, which is computationally expensive. In this paper, we propose to simulate roll-outs, exploiting the latest breakthroughs in image-to-image transduction, namely Pix2Pix GANs, to predict the dynamics of the environment. Our proposed algorithm, generative adversarial tree search (GATS), simulates rollouts up to a specified depth using both a GAN- based dynamics model and a reward predictor. GATS employs MCTS for planning over the simulated samples and uses DQN to estimate the Q-function at the leaf states. Our theoretical analysis establishes some favorable properties of GATS vis-a-vis the bias-variance trade-off and empirical results show that on 5 popular Atari games, the dynamics and reward predictors converge quickly to accurate solutions. However, GATS fails to outperform DQNs in 4 out of 5 games. Notably, in these experiments, MCTS has only short rollouts (up to tree depth 4), while previous successes of MCTS have involved tree depth in the hundreds. We present a hypothesis for why tree search with short rollouts can fail even given perfect modeling.
 This paper proposes to learn a dynamics model (state to pixels using Generative Adversarial Networks), use this model in conjunction with Monte Carlo Tree Search, model-free reinforcement learning (Q-learning) and a reward prediction network essentially combining model-free with model-based learning. The proposed approach is empirically evaluated on a small subset of the Atari games and theoretical analysis for the bias-variance tradeoff are presented. It is highly appreciated that this paper presents an idea and discusses why the proposed approach does not result in high performance. This is very valuable and useful for the community. On a high level it would be very useful if Figure 1 would be show less examples but present them much larger since it is almost impossible to see anything in a printout. Further, the caption does not lend itself to understand the figure. Similarly Figure 2 would benefit from a better caption. The first part of the discussion (7), the individual building blocks, should be mentioned much earlier in the paper. It would be further useful to add more related work on that abstraction level. This would help to communicate the main contribution of this paper very precisely. On the discussion of negative results: It is very interesting that Dyna-Q does not improve the performance and the hypothesis for why this is the case seems reasonable. Yet, it would be very useful to actually perform an experiment in a better controlled environment for which e.g. the dynamics model is based on the oracle and assess the empirical effect of different MCTS horizons and rollout estimates. Further, this scenario would allow to further quantify the importance and the required “quality” of the different learning blocks. In its current form the paper has theoretical contributions and experimental results which cannot be presented in the main paper due to space constraints. Albeit the appendix is already very extensive it would be very useful to structure it into the theoretical derivation and then one section per experiment with even more detail on the different aspects of the experiment. The story of the main paper would benefit from referencing the negative results more briefly and better analyzing the different hypothesis on toy like examples. Further, the introduction could be condensed in order to allow for more in detail explanations and discussions without repetition later on. As argued in the paper it is clear that image generation is a very expensive simulation mechanism which for games like pong which depend on accurate modeling of small aspects of the image are in itself difficult. Therefore, again although really appreciated, the negative results should be summarized in the main paper and the hypothesis concluded better analyzed. The extensive discussion of hyper parameters and approaches for individual components could be in the appendix and the main paper focuses on the hypothesis analysis.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Surprising Negative Results for Generative Adversarial Tree Search
While many recent advances in deep reinforcement learning rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment dynamics. One prospect is to pursue hybrid approaches, as in AlphaGo, which combines Monte-Carlo Tree Search (MCTS)—a model-based method—with deep-Q networks (DQNs)—a model-free method. MCTS requires generating rollouts, which is computationally expensive. In this paper, we propose to simulate roll-outs, exploiting the latest breakthroughs in image-to-image transduction, namely Pix2Pix GANs, to predict the dynamics of the environment. Our proposed algorithm, generative adversarial tree search (GATS), simulates rollouts up to a specified depth using both a GAN- based dynamics model and a reward predictor. GATS employs MCTS for planning over the simulated samples and uses DQN to estimate the Q-function at the leaf states. Our theoretical analysis establishes some favorable properties of GATS vis-a-vis the bias-variance trade-off and empirical results show that on 5 popular Atari games, the dynamics and reward predictors converge quickly to accurate solutions. However, GATS fails to outperform DQNs in 4 out of 5 games. Notably, in these experiments, MCTS has only short rollouts (up to tree depth 4), while previous successes of MCTS have involved tree depth in the hundreds. We present a hypothesis for why tree search with short rollouts can fail even given perfect modeling.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes to learn a dynamics model (state to pixels using Generative Adversarial Networks), use this model in conjunction with Monte Carlo Tree Search, model-free reinforcement learning (Q-learning) and a reward prediction network essentially combining model-free with model-based learning. The proposed approach is empirically evaluated on a small subset of the Atari games and theoretical analysis for the bias-variance tradeoff are presented. It is highly appreciated that this paper presents an idea and discusses why the proposed approach does not result in high performance. This is very valuable and useful for the community. On a high level it would be very useful if Figure 1 would be show less examples but present them much larger since it is almost impossible to see anything in a printout. Further, the caption does not lend itself to understand the figure. Similarly Figure 2 would benefit from a better caption. The first part of the discussion (7), the individual building blocks, should be mentioned much earlier in the paper. It would be further useful to add more related work on that abstraction level. This would help to communicate the main contribution of this paper very precisely. On the discussion of negative results: It is very interesting that Dyna-Q does not improve the performance and the hypothesis for why this is the case seems reasonable. Yet, it would be very useful to actually perform an experiment in a better controlled environment for which e.g. the dynamics model is based on the oracle and assess the empirical effect of different MCTS horizons and rollout estimates. Further, this scenario would allow to further quantify the importance and the required “quality” of the different learning blocks. In its current form the paper has theoretical contributions and experimental results which cannot be presented in the main paper due to space constraints. Albeit the appendix is already very extensive it would be very useful to structure it into the theoretical derivation and then one section per experiment with even more detail on the different aspects of the experiment. The story of the main paper would benefit from referencing the negative results more briefly and better analyzing the different hypothesis on toy like examples. Further, the introduction could be condensed in order to allow for more in detail explanations and discussions without repetition later on. As argued in the paper it is clear that image generation is a very expensive simulation mechanism which for games like pong which depend on accurate modeling of small aspects of the image are in itself difficult. Therefore, again although really appreciated, the negative results should be summarized in the main paper and the hypothesis concluded better analyzed. The extensive discussion of hyper parameters and approaches for individual components could be in the appendix and the main paper focuses on the hypothesis analysis.
assistant
 This paper proposes to learn a dynamics model (state to pixels using Generative Adversarial Networks), use this model in conjunction with Monte Carlo Tree Search, model-free reinforcement learning (Q-learning) and a reward prediction network essentially combining model-free with model-based learning. The proposed approach is empirically evaluated on a small subset of the Atari games and theoretical analysis for the bias-variance tradeoff are presented. It is highly appreciated that this paper presents an idea and discusses why the proposed approach does not result in high performance. This is very valuable and useful for the community. On a high level it would be very useful if Figure 1 would be show less examples but present them much larger since it is almost impossible to see anything in a printout. Further, the caption does not lend itself to understand the figure. Similarly Figure 2 would benefit from a better caption. The first part of the discussion (7), the individual building blocks, should be mentioned much earlier in the paper. It would be further useful to add more related work on that abstraction level. This would help to communicate the main contribution of this paper very precisely. On the discussion of negative results: It is very interesting that Dyna-Q does not improve the performance and the hypothesis for why this is the case seems reasonable. Yet, it would be very useful to actually perform an experiment in a better controlled environment for which e.g. the dynamics model is based on the oracle and assess the empirical effect of different MCTS horizons and rollout estimates. Further, this scenario would allow to further quantify the importance and the required “quality” of the different learning blocks. In its current form the paper has theoretical contributions and experimental results which cannot be presented in the main paper due to space constraints. Albeit the appendix is already very extensive it would be very useful to structure it into the theoretical derivation and then one section per experiment with even more detail on the different aspects of the experiment. The story of the main paper would benefit from referencing the negative results more briefly and better analyzing the different hypothesis on toy like examples. Further, the introduction could be condensed in order to allow for more in detail explanations and discussions without repetition later on. As argued in the paper it is clear that image generation is a very expensive simulation mechanism which for games like pong which depend on accurate modeling of small aspects of the image are in itself difficult. Therefore, again although really appreciated, the negative results should be summarized in the main paper and the hypothesis concluded better analyzed. The extensive discussion of hyper parameters and approaches for individual components could be in the appendix and the main paper focuses on the hypothesis analysis.