UNIVERSAL SUCCESSOR FEATURES APPROXIMATORS
The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs). Another way to generalise to new tasks is to exploit structure in the RL problem itself. Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs). Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI. We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical benefits and transfer abilities on a large-scale domain in which the agent has to navigate in a first-person perspective three-dimensional environment.
Paper’s contributions:
This paper considers the challenging problem of generalizing well to new RL tasks, based on having learned on a set of previous related RL tasks.  It considers tasks that differ only in their reward function (assume the dynamics are identical), and where the reward functions are constrained to be linear combinations over a set of given features.  The main approach, Universal Successor Features Approximators (USFAs) is a combination of two recent approaches:  Universal Value Function Approximators (UVFAs) and Generalized Policy Improvement (GPI).  The main claim is that while each of these methods leverages different types of regularity when generalizing to new tasks, USFAs are able to jointly leverage both types (and elegantly have both other methods as special cases).
Summary of evaluation:
Overall the paper tackles an important problem, and provides careful explanation and reasonably extensive results showing the ability of USFA to leverage structure.  I’m on the fence because I really wish the combination of generalization properties could be understood in a more intuitive way.  There are some more minor issues, such as lack of complexity analysis and a few notation details, that can be easily fixed.
Pros:
-	The problem of generalizing to new tasks in RL is an important open problem.
-	The paper is carefully written and provides clear explanation of most of the methods & results.
Cons:
-	The authors are diligent about trying to explain what type of regularities are exploited by each of UVFAs and GPI, and how this can be combined in USFAs.  However despite reading these parts carefully, I could not get a really good intuition, either in the methods or in the results, for the nature of the regularities exploited, and how it really differs.  Top of p.4 says that GPI generalizes well when the policy \pi(s) does well on task w’.  Can you give a specific MDP where Q is not smooth, but the policy does well?
-	There is no complexity analysis.  I would like to know the computational complexity of each of the key steps in Algorithm 1 (with comparison to simple UVFA and GPI).
-	It would be useful to see the empirical comparison with the approach of Ma et al. (2018), which also combines SFs and UFVAs. I understand there are differences in the details, but I would like to see confirmation of whether the claims about USFA’s superior ability to exploit structure is supported by results.
Minor comments:
-	The limitation to linear rewards is a reasonably strong assumption.  It would be good to support this, e.g. by references to domain that meet this assumption.
-	It seems the mathematical properties in Sec.3.1 could be further developed.
-	P.4: “Given a deterministic policy \pi, one can easily define a reward function r_\pi”.  I did not think this mapping was unique (see the literature on IRL, e.g. Ross et al.).  Can you provide a proof or reference to support this statement?
-	The definition of Q(s,a,w,z) is interesting. Can this be seen as a kernel between w and z?
-	\theta suddenly shows up in Algorithm 1. I presume these are the parameters of Q?  Should be defined.
-	The distribution used to sample policies seems to be a key step of this approach, yet not much guidance is given on how to do this in general.