Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn predictors for value from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future but have to compose with a potentially weak scalar signal (an estimate of the return). In this paper we develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end we determine which features of the future trajectory provide useful information to predict the associated return. This provides us with tractable prediction targets that are directly relevant for a task, and can thus accelerate learning of the value function. The idea can be understood as reasoning, in hindsight, about which aspects of the future observations could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale in challenging domains, including on 57 Atari 2600 games.
This paper presents a new model-based reinforcement learning method, termed hindsight modelling. The method works by training a value function which, in addition to depending on information available at the present time is conditioned on some learned embedding of a partial future trajectory. A model is then trained to predict the learned embedding based on information available at the current time-step. This predicted value is fed in place of the actual embedding to the same value model, to generate a value prediction for the current time-step. So instead of just learning a value function based on future returns, the method uses a two-step process of learning an embedding of value relevant information from the future and then learns to predict that embedding.
The paper gives some motivating examples of why and when such an approach could yield an advantage over standard value learning methods like Monte-Carlo or temporal difference learning. The basic idea is that when the returns obey some causal structure like X->Y->Z it may be easier to learn P(Y|X) and P(Z|Y) than to learn P(Z|X) directly. In particular, the authors point out that in the discrete case when Y takes relatively few values the size of the respective probability tables can be smaller in the former case than the latter. This motivates the approach of discovering a set of future variables to predict which are themselves predictive of return, rather than predicting the expected future return directly.
On a high level, I like the idea of specifically learning to model relevant aspects of the environment. However, I lean toward rejecting this work in its current form because I feel the motivation for when this particular method would be useful is unclear.
In particular, I don't really understand how this method could, in general, be expected to improve on regular bootstrapping. Why learn a prediction of the return at time t based on future information when we could just use the value function at a later time to improve the prediction at time t? It seems to me that the future value function itself concisely summarizes the information in the future state that is relevant for predicting the past return, while better exploiting the structure of the problem. Of course in cases like partial observability, it could be that the future value function lacks information from the past that is important for accurately predicting the return (for example in the portal example of this paper). However, if partial observability is really the case of interest, the method presented in this paper seems like a rather roundabout solution method. For example, instead of conditioning v+ on a future hidden state h_{t+k} (as the authors do in the experiments) perhaps one could simply condition the value function on a past hidden state h_{t-k} and obtain similar benefit from bootstrapping?
Aside from partial observability, for which I feel there are better approaches, the only situation I can understand the method having an advantage is when later states contain information which helps to predict earlier rewards. This is essentially the situation presented in the illustrative example. However, currently I feel such situations are rather contrived and unintuitive so I would need more supporting evidence to accept these situations as a good motivation.
On a deeper level, I don't see how the probability table motivation given in the introduction applies when what is being learned is an expectation (i.e. a value function) and not a distribution.
The approach also suffers from well-known issues with using the output of an expectation model of a variable as the input to a nonlinear function approximator in place of the variable itself. Namely, there is no guarantee that the expectation value of a variable is a possible value for the variable so giving it as input to a predictor trained on the variable itself could easily yield nonsense output in the stochastic case. As far as I can tell the method does nothing to mitigate this (please correct me if I'm wrong), so there is no reason to assume the method is generally applicable in settings with nontrivial stochasticity.
Despite these concerns, I feel the experiments are for the most part quite well thought out and executed. The paper is also quite well written, motivation issues aside, so I would not be upset if it was accepted with the hope that it leads to future work addressing the above-mentioned concerns.
If possible I think this paper would benefit significantly from a detailed explanation of how and when the proposed approach should be expected to improve on bootstrapping, including bootstrapping off a value function which uses an analogous architecture to v+.
Questions for authors:
Given the hyper-parameters of R2D2 deviate somewhat from those used in the original paper, and nothing is said about how they were chosen, how confident can we be that the observed advantage of hindsight modelling is not simply due to hyper-parameters being selected which are more favourable for the proposed method?
Given that you are not learning distributions but expectations in the form of value functions, how pertinent is the motivation of learning P(Y|X) and P(Z|Y) instead of P(Z|X) directly described in the introduction?
How much of the benefit observed in the portal example an ATARI could also be gained from simply providing the value function approximation with h_{t-k} as input to help span larger time-gaps?
Update:
While I still feel the exposition could be improved to make the underlying idea clearer, I feel the authors did a good job of addressing my major concerns in their reply, hence I have raised my score to a weak accept.
I have to admit I missed the point that v^+ and v^m were using entirely different parameter sets. In light of this, I agree that the expectation model issue I mentioned is not a major concern.
I also appreciate the clarification of the hyperparameters, if they were really tuned to improve the baseline then this detail should be added to the paper and would negate my concern there. 
Finally, I thank the authors for providing the value-function oriented example. I found this example to be more illustrative than the one in the introduction of the paper, and I now feel that I have a better grasp of the motivation. I still have doubts about the general benefit of the approach over bootstrapping but since it is not entirely clear to me one way or the other I feel the idea at least warrants further exploration, and it would be reasonable to accept the paper to make the community aware of it.