Value Prediction Network
This paper proposes a novel deep reinforcement learning (RL) architecture, called Value Prediction Network (VPN), which integrates model-free and model-based RL methods into a single neural network. In contrast to typical model-based RL methods, VPN learns a dynamics model whose abstract states are trained to make option-conditional predictions of future values (discounted sum of rewards) rather than of future observations. Our experimental results show that VPN has several advantages over both model-free and model-based baselines in a stochastic environment where careful planning is required but building an accurate observation-prediction model is difficult. Furthermore, VPN outperforms Deep Q-Network (DQN) on several Atari games even with short-lookahead planning, demonstrating its potential as a new way of learning a good state representation.
This paper proposes a new learning architecture, VPN, whose components can be
learned end-to-end to perform planning over pre-specified temporally extended actions. An advantage of this approach is that it explicitly avoids working in the original observation space and plans indirectly over a learned representation.
This is made possible by a applying a model-free DQN-like approach for learning values which are then used in combination with reward and duration models of options
to form planning targets. The proposed approach is shown to outperform DQN in a
variety of Atari games.
Model-based RL is a difficult problem when it comes to a large domain like ALE.
The development of new model-based methods which do not explicitly try to
reconstruct/predict the next observation seem to be going in the right direction.
VPN is a contribution in this new line of research.
However, there are a few conceptual elements which remain unclear.
First, why predicting the expected discount/duration separately from the
option-conditional expected cumulative return to termination ? Using the
Bellman-like equations for the reward model of options (see equation 17 of the options paper), the discount factor can effectively be folded into the reward prediction. This would avoid the need to learn a separate discount prediction for each option.
Second : The nature of the "next abstract state" (line 109) is a key element of the
proposed approach to avoid learning a transition model directly
in the original space. As you mention later on line 251, there is no incentive/constraint for the learned state representation to capture useful properties for planning and could in the extreme case become a tabular representation. The appendix then shows that you use a particular architecture for the transition module which predicts the difference. How much this prior on the architecture affects quality of the representation learned by the system ? What kind of representation would be learned in the absence of a residual connection ?
Can you also elaborate on the meaning of the weighting (1/d, d-1/d)  in equation (1) ? Why is it necessary ? Where does that come from ?
Line 21 : "predict abstractions of observations"
This expression is unclear, but I guess that it means a feature/representation
of the state. You also write : "not clear how to roll such predictions", but isn't that exactly what VPN is doing ?
Typo : line 31 "dyanmics"