Transfer of Value Functions via Variational Methods
We consider the problem of transferring value functions in reinforcement learning. We propose an approach that uses the given source tasks to learn a prior distribution over optimal value functions and provide to an efficient variational approximation of the corresponding posterior in a new target task. We show our approach to be general, in the sense that it can be combined with complex parametric function approximators and distribution models, while providing two practical algorithms based on Gaussians and Gaussian mixtures. We theoretically analyze them by deriving a finite-sample analysis and provide a comprehensive empirical evaluation in four different domains.
Update: ----------- I had a look at the author response: It seems reasonable, contains a lot of additional information / additional experiments which do address my main concerns with the paper. Had these comparisons been part of the paper in the first place I would have voted for accepting the paper.  I am now a bit on the fence about this as the paper could be accepted but will require a major revision, I will engage in discussion with the other reviewers and ultimately the AC has to decide whether such big changes to the experimental section are acceptable within the review process.   Original review: --------------------- The paper presents a method for transfer learning via a variational inference formulation in a reinforcement learning (RL) setting.  The proposed approach is sound, novel and interesting and could be widely applicable (it make no overly restrictive assumptions on the form of the learned (Q-)value function).  The paper is well written and the used terminology is introduced clearly. The paper is set into contextclearly and points out connections to existing theoretical work where appropriate. I quite enjoyed reading the theoretical section.  The proposed solution for the transfer learning approach is simple and appealing, albeit a few open problems remain in my head as to how well the presented approach can work in scenarios where the overlap between target and source domains is small (as the method makes no attempt to directly optimized for learning speed on the target domain).  However, two main concerns remain regarding the empirical evaluation of the proposed approach (see below). Frankly, I am dissapointed how dismissive the authors are of any potential comparison to existing approaches! As a result no clear assesment can be made regarding the usefulness of the presented approach - although the theoretical derivation might be of separate interest (albeit limited in it's use, see comment below). This failure pulls the paper clearly below the threshold for acceptance for me.  Main concerns: 1) The paper does not consider the influence of the initial weights on the convergence speed of the target task (which is e.g. considered in recent meta-learning algorithms). While the approach is sound as presented it begs the question: how would it compare to an approach that is more directly tailored to finding Q-function/policy-parameterizations that are easy to adapt ? The fact that the experimental section makes no attempt at answering this question and, instead, simply dismisses all existing approaches for transfer learning as "not-comparable" without giving any further explanation is concerning! For a NIPS paper this is simply not good enough. At least simple baselines like transferring from a random source task and fine-tuning need to be provided, especially when the considered domains are so simple that running an individual experiment takes on the order of minutes rather than hours. 2) Aside from the main concern described in 1 the considered domains are extremely simple and have huge overlap. State of the art transfer learning approaches nowadays consider more complicated domains (see e.g. [1]). While this is not a major concern given that the approach is interesting enough and novel it does make me wonder whether the method is limited in its applicability.  Other remarks: - Section 3.4 nicely states - and partially addresses - a question I had in my head: "why and how is it valid to use the only sub-differentiable TD-error as the error function for this approach". While this concern is stated precisely - and subsequently addressed by moving towards a mellowmax style objective (which should potentially be mentioned earlier) - the section fails to exactly pinpoint why using the TD-error would not work here, and no experimental or theoretical confirmation is given thereof, we are only presented with a suggestive argument. I would have liked to see more here. - Theorem 1 might be of independent interest and is explained nicely in the main paper. - Theorem 2 is of little practical use as one cannot really assume linearity in the adaptable parameters (unless the distribution over tasks is quite restrictive). Even if the features \phi are expressive (i.e. features extracted from a neural network or the like) transfer via simply adapting the linear weights cannot be expected to work in general (and I believe [1,2] already show examples for failure cases).   Concerns regarding the experiments: 1) I would dispute the claim made in the beginning of the experiments section that: "To the best of our knowledge, no existing transfer algorithm is directly comparable to our approach from an experimental perspective" (l. 254). Sure, no existing algorithms attempts to perform transfer as done in this paper, but clearly you could attempt to transfer via successof features or MAML or any other transfer learning method for RL in the same way: train on whatever data you give to your approach, then evaluate give as much data as collected by your approach on the target domain and see how well transfer occurs. In fact similar domains to your maze navigation and other experiments have been studied in the literature see [1-4]. 2) Similar to point 1: your domains in the transfer experiments overlap heavily! You need to provide a baseline for how well a simple fine-tuning approach (start with parameters from 1 random source domain, fine-tune on target task) would work. I suspect that this would work quite well on the Montain Car task. 3) How can 1-MGVT ever be better than GVT ? Clearly given the parametrization from 3.2 and 3.3 the results should be exactly the same ? The only thing I can explain this result is that you mean 1 component per source task, but then use a mixture over all source tasks for the target task. In which case this should be stated clearly. 4) Are the results averaged over multiple runs or merely wrt. different start positions ? What do the graphs even show ? Mean performance ? Median performance ? 1 std. deviation or 2 or .... ?  Minor errors: l. 24: "and so on" remove this, unecessary. l. 27: "to attempt at" -> "to attempt to characterize" l. 44: "Leveraging on" -> remove on l. 60: is policy deterministic here ? l. 138: "which task *it* is solving* l. 201: proved -> proven l. 203: "such *a* fixed point" l. 236: "in such *a* case" l. 313: "from such a set" Why does the supplementary contain the main paper ? That should be fixed.  Reviewer remark: I have no verified the proof of Theorem 2.  [1] Successor features for transfer in reinforcement learning. André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver [2] Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments. Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, Wolfram Burgard [3] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. Chelsea Finn, Pieter Abbeel, Sergey Levine [4] Unsupervised Meta-Learning for Reinforcement Learning. Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, Sergey Levine