We introduce Dynamic Planning Networks (DPN), a novel architecture for deep reinforcement learning, that combines model-based and model-free aspects for online planning. Our architecture learns to dynamically construct plans using a learned state-transition model by selecting and traversing between simulated states and actions to maximize valuable information before acting. In contrast to modelfree methods, model-based planning lets the agent efficiently test action hypotheses without performing costly trial-and-error in the environment. DPN learns to efficiently form plans by expanding a single action-conditional state transition at a time instead of exhaustively evaluating each action, reducing the required number of state-transitions during planning by up to 96%. We observe various emergent planning patterns used to solve environments, including classical search methods such as breadth-first and depth-first search. DPN shows improved data efficiency, performance, and generalization to new and unseen domains in comparison to several baselines.
This paper proposes a new architecture for model-based deep RL, in which an “inner agent” (IA) takes several planning steps to inform an “outer agent” (OA) which actually acts in the world. The main contributions are to propose a new objective for the IA, and to allow the IA to “undo” its imagined actions. Overall I think this could be a great paper, but it needs further justification of some of the architectural choices and more rigorous analysis/experiments before it will be ready for acceptance.
Pros:
- Nice demonstration of improved data efficiency over existing model-based methods.
- Substantial improvement over other model-based methods in terms of computational cost.
- Interesting qualitative analysis showing discovery of DFS and BFS-like search procedures.
Cons:
- Limited novelty over existing methods.
- It is unclear what in the model contributes to improved performance.
Quality
---------
The results in the paper seem impressive in terms of sample complexity, but I think there needs to be further exploration of the source of the results. I strongly suggest including in a revision a number of ablation experiments to tease these details apart---for example, what do the results look like if the IA uses the same objective as the OA? Does the agent achieve worse performance if it has to restart its imaginations from the root of the tree each time, as is more analogous to MCTS and other previous model-based approaches?
Additionally, there are a few places in the paper where unjustified statements are made. For example, in Section 5.3, the paper states that “we hypothesize that focusing, by repeatedly visiting the same state, the IA ensures that the POI is remembered in its hidden state such that the OA can act accordingly, given this information”. This seems very speculative. It would certainly be very interesting if true, but there needs to be something more than just intuition to back up this hypothesis. I recommend including some probe experiments (e.g., force the IA to take a sequence of such actions, or not, and see what the result on the behavior of the OA is) or removing speculations such as this (or moving it to the appendix).
The literature review is missing some related work, particularly from the realm of model-based continuous control. [1-3] are a few references to start with; these papers take a different approach in that they don’t use tree search but they are still worthwhile discussing. I think a reference to [4] is also missing, which takes a related approach to learning the decisions needed to perform MCTS.
Clarity
--------
Overall, the paper is well-written and I understand what was done and how the architecture works. However, I had a hard time understanding the choice of the inner objective (Equation 1). The paper states that this equation defines the “value of information” and defines it as the product of the KL from the OA’s hidden state prior to the transition to after the transition, multiplied by the Q value estimated by the OA and the action probability of the IA. This is very mysterious to me. Why is this a good objective? Why does the KL term of the hidden state of the OA have anything to do with the value of information? Given that the difference in objective of the IA is one of the main contributions of the paper, this choice needs to be justified, explained, and examined. As mentioned above, it would be best if a revision could include some ablation experiments where the choice of this objective is more closely examined.
More broadly, as mentioned above, it is unclear to me what part of the framework results in improved performance. Is it the ability to “undo” actions (rather than starting over from the root or exhaustively performing BFS), or is it the KL-based reward given to the IA? The paper does not provide any insight into this question, making it unclear what are the key points I should take away. 
Minor:
- The colors in the caption of Figure 5 do not match the colors in the figure.
- The colors in Figure 7 are very dark and it is hard to make out what is actually happening in the figure.
Originality
-------------
The objective of the inner agent (Equation 1) appears novel, though as discussed above it is unclear to me what exactly it means and what its implications are. The idea of constructing an imagination tree state-by-state is not particularly novel, and was previously explored by Pascanu et al. (2017). I think this paper deserves more discussion and comparison than it is given in the present work (in particular, compare Figure 3 of the present paper and Figure 2 of Pascanu et al.). In general, the main idea in both papers is the same: have an agent learn to take internal planning steps and construct a planning tree that then informs the final action in the world. The biggest differences from Pascanu et al. are that the present work uses a separate objective for the inner agent, and allows taking a step backwards and returning to the previous state (whereas Pascanu et al. only allowed imagining from the current imagined state or from the root). So, the overall the paper has some new ideas, but is not highly novel compared to previous work. I see the two biggest original contributions as being: (1) the separate objective in the inner agent and (2) the ability for the agent to restart its imagination from the previous imagined state.
Significance
----------------
The results reported by the paper are significant in that they do show dramatic improvement in sample complexity over existing model-free methods, as well as improvement in computational cost over existing model-based methods. However, as discussed above, it is hard for me to know what conclusions I should draw from the paper in terms of what aspects of the approach drive this performance. Thus, I think the lack of clarity in this respect limits the significance of the paper.
[1] Finn, C., & Levine, S. (2017). Deep visual foresight for planning robot motion. In Proceedings of the International Conference on Robotics and Automation (ICRA 2017).
[2] Srinivas, A., Jabri, A., Abbeel, P., Levine, S., & Finn, C. (2018). Universal planning networks. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018).
[3] Henaff, M., Whitney, W., & LeCun, Y. (2018). Model-based planning with discrete and continuous actions. arXiv preprint arXiv:1705.07177
[4] Guez, A., Weber, T., Antonoglou, I., Simonyan, K., Vinyals, O., Wierstra, D., … Silver, D. (2018). Learning to search with MCTSnets. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018).