Transfer in Reinforcement Learning (RL) refers to the idea of applying knowledge gained from previous tasks to solving related tasks. Learning a universal value function (Schaul et al., 2015), which generalizes over goals and states, has previously been shown to be useful for transfer. However, successor features are believed to be more suitable than values for transfer (Dayan, 1993;Barreto et al., 2017), even though they cannot directly generalize to new goals. In this paper, we propose (1) Universal Successor Features (USFs) to capture the underlying dynamics of the environment while allowing generalization to unseen goals and (2) a flexible end-to-end model of USFs that can be trained by interacting with the environment. We show that learning USFs is compatible with any RL algorithm that learns state values using a temporal difference method. Our experiments in a simple gridworld and with two MuJoCo environments show that USFs can greatly accelerate training when learning multiple tasks and can effectively transfer knowledge to new tasks.
Summary: This paper proposes a generalisation of the SFs framework to a goal conditioning representation that could, in principle, generalise over a collection of goals at test time. This is akin to universal value functions [1] (and more generally GVFs). Although I like the idea  and it seems a very interesting direction for generalisation to new goals, I do think the execution, the particular instantiation and (lack of) in-depth evaluation with (at least some of the) existing methods in literature -- including UVFAs [1] and the different ways SFs have been used for generalisation [2,3,4] -- is unfortunately letting it down.
Clarity: Reasonably well-written, easy to follow. A couple of things in the experimental section can be improved:
- It’s not totally clear to me what the their baseline Multi-goal DQN is. Does it have the same architecture as Figure 1, but just using (2).
- In the plots, the only difference between DQN and DQN+USF is that the second as the additional loss L_{\psi} ? Or is there any other difference? 
Originality and Significance: 
I’m a bit split here: I like in principle the idea, but I think this instantiation is (fairly) incremental with respect to the current literature. Even the claimed contributions are a bit thin. Suitability of SFs to any TD-based learning, comes from SRs/SFs satisfying a Bellman eq. which was point out, explored and paired with control algorithms before [2,4]. Also, the particular way of learning the features \phi, without going through the rewards, was already proposed and explored in [3]. That might be a missing reference. 
The experiments seems to show slight improvements with respect to a baseline (Multi-DQN). It is not clear to me exactly what this is or if it would dominated even something vanilla UVFAs. I think this is a missing and somewhat mandatory comparison. I know the authors noted that is was because ‘UVFAs are prone to instabilities and may require further prior knowledge’, but I think that refers only to the two-stage (factorisation) procedure proposed in the original paper, not the common adoption in the literature. At the end of the day, the proposed architecture in Fig. 1 is a kind of UVFA, just with a bit more structure, so it would be surprising to me if UVFAs would actually fail in these environments. But if that’s the case, that’s a very interesting data point that the additional structure actually helps considerably beyond the incremental advantage exemplified here. 
Other comments/questions:
1) Clarification on the training procedure. The value function $Q(s,a,g)$  are training via eq. (3) with the i) actual reward (coming from the environment) or ii) the ‘fictitious’ reward coming from r(s,a,s’|g) = \phi(s,a,s’)^T w(g)? Note that these are very different and only one ensures compatibility between the rewards and the value functions in learning.
The SFs will give you the value function for the reward r(s,a,s’|g) = \phi(s,a,s’)^T w(g) and if this is not align with the real reward, the corresponding value function obtained via SFs will not be the value function optimising the real reward. As far as I can see there’s not criteria that forces this to be the case.
2) Comparison with SF transfer literature. Although discussed in the related work section, there is no quantitative comparison to the way SFs were shown to transfer knowledge[2,4], via evaluation and (generalised) policy improvement. Because these ways of generalisation are very different, it’s not clear go they would stack against each other, or in which scenarios one would be more appropriate than the other.
To give a more concrete example: The training procedure in 3.1 makes sure that there’s fairly good coverage of the whole state-space by sampling goals conditioned on the room. Now if one would train SFs on these train tasks only (even independently), we would have policies that would know how to go to any of the rooms. And for the test tasks we would have the evaluation of these policies to the collection of goals. Which means that applying the methodology of transfer in [2,4] we would zero-shot get policies that reach any of the states encountered in the path of the 12 goals used in the train phase. And even if the test goals are not part of this collection, it stands to reason that a policy that can already go to the goal’s room and be easily adaptable to reaching the test goal -- aka the evaluation the policy that already reached that room is a good starting point for the improvement step [4].
Note: I am willing to reconsider when/if the above have been reconciled/resolved.
References:
[1] Schaul, T., Horgan, D., Gregor, K. and Silver, D., 2015, June. Universal value function approximators. In International Conference on Machine Learning (pp. 1312-1320).
[2] Andre Barreto, Will Dabney, Remi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and ´ David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4055–4065, 2017.
[3] Machado, M.C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G. and Campbell, M., 2018. Eigenoption Discovery through the Deep Successor Representation, International Conference on Learning Representations, 2018.
[4] Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A. and Munos, R., 2018, July. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning (pp. 510-519).