While many recent advances in deep reinforcement learning rely on model-free methods, model-based approaches remain an alluring prospect for their potential to exploit unsupervised data to learn environment dynamics. One prospect is to pursue hybrid approaches, as in AlphaGo, which combines Monte-Carlo Tree Search (MCTS)-a model-based method-with deep-Q networks (DQNs)-a model-free method. MCTS requires generating rollouts, which is computationally expensive. In this paper, we propose to simulate roll-outs, exploiting the latest breakthroughs in image-to-image transduction, namely Pix2Pix GANs, to predict the dynamics of the environment. Our proposed algorithm, generative adversarial tree search (GATS), simulates rollouts up to a specified depth using both a GANbased dynamics model and a reward predictor. GATS employs MCTS for planning over the simulated samples and uses DQN to estimate the Q-function at the leaf states. Our theoretical analysis establishes some favorable properties of GATS vis-a-vis the bias-variance trade-off and empirical results show that on 5 popular Atari games, the dynamics and reward predictors converge quickly to accurate solutions. However, GATS fails to outperform DQNs in 4 out of 5 games. Notably, in these experiments, MCTS has only short rollouts (up to tree depth 4), while previous successes of MCTS have involved tree depth in the hundreds. We present a hypothesis for why tree search with short rollouts can fail even given perfect modeling.
The submitted paper proposes GATS, a RL model combining model-free and model-bases reinforcement learning. Estimated models of rewards and dynamics are used to perform rollouts in MTCS without actually interacting with the true environment. The authors theoretically and empirically evaluate their approach for low depths rollouts. Empirically they show improved sample complexity on the Atari game Pong.
I think publishing negative research results is very important and should be done more often if we can learn from those results. But that is an aspect I feel this paper falls short at. I understand that the authors performed a great deal of effort trying to tune their model and evaluated many possible design choices, but they do not a provide a thorough investigation of the causes which make GATS "fail". I suggest that the authors try to understand the problems of MCTS with inaccurate models better with synthetic examples first. This could give insights into what the main sources of the problem are and how they might be circumvented. This would make the negative results much more insightful to the reader as each source of error is fully understood (e.g., beyond an error reate for predicting rewards which does not tell us about the distribution of errors which for example could have a big effect on the author's observations).
Another issue that needs further investigation is the author's "hypothesis on negative results". It would be great to experimentally underline the author's arguments. It is not trivial (at least to me) to fully see the "expected" interaction of learning dynamics and depths of rollouts. While MCTS should be optimal with any depths of rollouts given the true Q-function, the learning process seems more difficult to understand.
I would also like the authors to clarify one aspect of their theoretical analysis. e_Q is defined as the error in the Q-estimate for any single state and action in the main text. This appears to be inconsistent with the proof in the appendix, making the bound miss a factor which is exponential in H (all possible x_H that can be reached within H steps). This would change the properties of the bound quite significantly. Maybe I missed something, so please clarify.
Originality mainly comes from the use of GANs in MCTS and the theoretical analysis.
Strengths:
- Interesting research at the intersection of model-free and model-based research
- Lots of effort went into properly evaluating a wide range of possible design choices
- Mainly well written
Weaknesses:
- Missing depths in providing a deep understanding of why the author's expectations and empirical findings are inconsistent
- The authors use many tweaks and ideas to tune each component of their model making it difficult to draw conclusions about the exact contribution of each of these
- Error in the theoretical analysis (?)
Minor comment:
* The paper would benefit from improved and more self-contained figure captions.