Published as a conference paper at ICLR 2018 TEMPORAL DIFFERENCE MODELS: MODEL-FREE DEEP RL FOR MODEL-BASED CONTROL
Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based RL methods. Our experimental results show that, on a range of continuous control tasks, TDMs provide a substantial improvement in efficiency compared to state-of-the-art model-based and model-free methods.
This paper proposes a "temporal difference model learning", a method that aims to combine the benefits of model-based and model-free RL.  The proposed method essentially learns a time-varying goal-conditional value function for a specific reward formulation, which acts as a surrogate for a model in an MPC-like setting.  The authors show that the method outperforms some alternatives on three continuous control domains and real robot system.
I believe this paper to be borderline, but ultimately below the threshold for acceptance.  On the positive side, there are certainly some interesting ideas here: the notion of goal-conditioned value functions as proxies for a model, and as a means of merging model-free and model-based approaches is very really interesting, and hints at a deeper structure to goal-conditioned value functions in general.  Ultimately, though, I feel that there are two main issues that make this research feel as though it is still ultimately in the earlier stages: 1) the very large focus on the perspective that this approach is unifying model-based and model-free RL, when it fact this connection seems a bit tenuous; and 2) the rather lackluster experimental results, which show only marginal improvement over purely model-based methods (at the cost of much additional complexity), and which make me wonder if there's an issue with their implementation of prior work (namely the Highsight Experience Replay algorithm).
To address the first point, although the paper stresses it to a very high degree, I can't help but feel that the connection that the claimed advance of "unifying model-based and model-free RL" is overstated.  As far as I can tell, the connection is as follows: the learned quantity here is a time-varying goal-conditioned value function, and under some specific definition of reward, we can interpret the constraint that this value function equal zero as a proxy for the dynamics constraint in MPC.  But the exact correspondence between this and the MPC formulation only occurs for a horizon of size zero: longer horizons require a multi-step MPC for the definition of the model-free and model-based correspondence.  The fact that the action selection of a model-based method and this approach have some function which looks similar (but only under certain conditions), just seems like a fairly odd connection to highlight so heavily.
Rather, it seems to me that what's happening here is really quite simple: the authors are extending goal-conditioned value functions to the case of non-stationary finite horizon value functions (the claimed "key insight" in eq (5) is a completely standard finite-horizon MDP formulation).  This seems to describe perfectly well what is happening here, and it does also seem intuitive that this provides an advantage over stationary goal-conditioned value functions: just as goal conditioned value functions offer the advantage of considering "every state as a goal", this method can consider "every state as a goal for every time horizon".  This seems interesting enough on its own, and I admit I don't see the need for the method to be yet another claimed unification of model-free and model-based RL.
I would also suggest that the authors look into the literature on how TD methods implicitly learn models (see e.g. Boyan 1997 "Least-squares temporal difference learning", and Parr et al., 2007 "An analysis of linear models...").  In these works it has been shown that least squares TD methods (at least in the linear feature setting), implicitly learn a dynamics model in feature space, but only the "projection" of the reward function is actually needed to learn the TD weights.  In building the proposed value functions, it seems like the authors are effectively solving for multiple rewards simultaneously, which would effectively preserve the learned dynamics model.  I feel like this may be an interesting line of analysis for the paper if the authors _do_ want to stick with the notion of the method as unifying model-free and model-based RL.
All these points may ultimately just be a matter of interpretation, though, if not for the second issue with the paper, which is that the results seem quite lackluster, and the claimed performance of HER seems rather suspicious.  But instead, the authors evaluate the algorithm on just three continuous control tasks (and a real robot, which is more impressive, but the task here is still so extremely simple for a real robot system that it really just qualifies as a real-world demonstration rather than an actual application).  And in these three settings, a model-based approach seems to work just as well on two of the tasks, and may soon perform just as well after a few more episodes on the last task (it doesn't appear to have converged yet).  And despite the HER paper showing improvement over traditional policy approaches, in these experiments plain DDPG consistently performs as well or better than HER.