Humans learn to play video games significantly faster than the state-of-the-art reinforcement learning (RL) algorithms. People seem to build simple models that are easy to learn to support planning and strategic exploration. Inspired by this, we investigate two issues in leveraging model-based RL for sample efficiency. First we investigate how to perform strategic exploration when exact planning is not feasible and empirically show that optimistic Monte Carlo Tree Search outperforms posterior sampling methods. Second we show how to learn simple deterministic models to support fast learning using object representation. We illustrate the benefit of these ideas by introducing a novel algorithm, Strategic Object Oriented Reinforcement Learning (SOORL), that outperforms state-of-the-art algorithms in the game of Pitfall! in less than 50 episodes.
-- Summary --
The paper proposes to learn (transition) models (for MDPs) in terms of objects and their interactions. These models are effectively deterministic and are compatible with algorithms for planning with count-based exploration. The paper demonstrates the performance of one such planning method in toy tasks and in Pitfall, as well as a comparison with other planning methods in the toy tasks. The proposed model-based method, called SOORL, yields agents that perform better on Pitfall with a small amount of data.
-- Assessment --
As a positive, the results of the paper are favorable compared to previous work, with good sample efficiency, and they demonstrate the viability of the proposed approach. The most negative point is that SOORL relies on limiting domain-specific biases that are hard to remove or circumvent.
-- Clarity --
The paper is somewhat clear. There are many typos and mistakes in writing, and at parts (for example, the second paragraph of Section 4.2) the explanations are not clear.
-- Originality --
I believe the work is original. The paper explores a natural idea and the claims/results are not surprising, but as far as I am aware it has not been tried before.
-- Support --
The paper provides support for some of the claims made. The comparison to related work contains unsupported claims ("we studied how imperfect planning can affect exploration") and could be more upfront about the weaknesses of the proposed method. The claims in the introduction are sufficiently supported.
-- Significance --
It would be hard to scale SOORL to other tasks, so it is unlikely to be adopted where end-to-end learning is wanted. Therefore I believe the impact of the paper to be limited.
There is also the question of whether the paper will attract interest and people will work on addressing the limitations of SOORL. I would like to hear more from the authors on this point.
-- For the rebuttal --
My greatest doubt is whether the paper will attract enough interest if published, and it would be helpful to hear from the authors on why they think future work will build on the paper. Why is the proposed approach a step in the right direction?
-- Comments --
Sample efficiency: The paper should be more clear about this point. It seems that 50 episodes were used for getting the positive reward in Pitfall, which is great.
Object detection: I am happy with the motivation about how we can remove the hand-made object detection. It is important the other strong assumptions (object interaction matrix, for example) can be removed as well. My opinion on simplifications is this: They are ok if they are being used to make experiments viable and they can be removed when scaling up; but they are not ok if there is no clear way to remove them.
Known interaction matrix: It may be possible to remove this requirement using the tools in [1]
Deterministic model: The use of no-ops to make the model deterministic seems right if the ultimate goal is to make the model deterministic, but it seems unsuited if the model is to be used for control. Maybe the model needs to be temporally extended as I thought the paper was proposing in Section 4.2 but Section 4.3 suggests that this temporal extension was not a good idea. Is my understanding correct?
Exploration: I was a bit confused about how the text discusses exploration. UCT uses OFU, but the text suggests that it does not. What are the components for exploration? Both a bonus on unseen transitions and the confidence interval bonus? Also, the paper would have to provide support for the claim that "with limited number of rollouts, the agent might not observe the optimistic part of the model, in contrast to optimistic MCTS where optimism is build into every node of the tree". However, it is fair to say that in the to domains MCTS seemed has performed better, and for that reason it has been chosen instead of Thompson Sampling for the later experiments.
Writing: The paper has a number of typos and mistakes that need to be fixed. To point out a few:
* I would suggest more careful use of "much" and "very"
* For citations, "Diuk et al. (2008) also proposed..." and "(UCT, Kocsis & Szepesvari, 2006)"
Claims: I think the claims made in the introduction could be stated more clearly in the conclusion. (Intro) "We show how to do approximate planning" --> (Conclusion) "Our model learning produces effectively deterministic models that can then be used by usual planning algorithms".
-- References --
[1] Santoro et al., 2017. "A simple neural network module for relational reasoning"