Published as a conference paper at ICLR 2018 UNIVERSAL AGENT FOR DISENTANGLING ENVIRONMENTS AND TASKS
Recent state-of-the-art reinforcement learning algorithms are trained under the goal of excelling in one specific task. Hence, both environment and task specific knowledge are entangled into one framework. However, there are often scenarios where the environment (e.g. the physical world) is fixed while only the target task changes. Hence, borrowing the idea from hierarchical reinforcement learning, we propose a framework that disentangles task and environment specific knowledge by separating them into two units. The environment-specific unit handles how to move from one state to the target state; and the task-specific unit plans for the next target state given a specific task. The extensive results in simulators indicate that our method can efficiently separate and learn two independent units, and also adapt to a new task more efficiently than the state-of-the-art methods.
The authors propose to decompose reinforcement learning into a PATH function that can learn how to solve reusable sub-goals an agent might have in a specific environment and a GOAL function that chooses subgoals in order to solve a specific task in the environment using path segments. So I guess it can be thought of as a kind of hierarchical RL.
The exposition of the model architecture could use some additional detail to clarify some steps and possibly fix some minor errors (see below). I would prefer less material but better explained. I had to read a lot of sections more than once and use details across sections to fill in gaps. The paper could be more focused around a single scientific question: does the PATH function as formulated help?
The authors do provide a novel formulation and demonstrate the gains on a variety of concrete problems taken form the literature. I also like that they try to design experiments to understand the role of specific parts of the proposed architecture.
The graphs are WAY TOO SMALL to read. Figure #s are missing off several figures.
MODEL & ARCHITECTURE
The PATH function given a current state s and a goal state s', returns a distribution over the best first action to take to get to the goal P(A).  ( If the goal state s’ was just the next state, then this would just be a dynamics model and this would be model-based learning? So I assume there are multiple steps between s and s’?). 
At the beginning of section 2.1, I think the authors suggest the PATH function could be pre-trained independently by sampling a random state in the state space to be the initial state and a second random state to be the goal state and then using an RL algorithm to find a path. 
Presumably, once one had found a path ( (s, a0), (s1, a1), (s2, a2), …, (sn-1,an-1),  s’ ) one could then train the PATH policy on the triple (s, s’, a0) ? This seems like a pretty intense process: solving some representative subset of all possible RL problems for a particular environment … Maybe one choses s and s’ so they are not too far away from each other (the experimental section later confirms this distance is >= 7. Maybe bring this detail forward)?
The expression Trans’( (s,s’), a) = (Trans(s,a), s’) was confusing.  I think the idea here is that the expression 
Trans’(  (s,s’) , a ) represents the n-step transition function and ‘a' represents the first action?
The second step is to train the goal function for a specific task. So I gather our policy takes the form of a composed function and the chain rule gives close to their expression in 2.2
    PATH(  s,  Tau( s, th^g ),  a ; th^p )
    d / { d th^g }  PATH(  s,  Tau( s, th^g ),  a ; th^p ) 
         =  {d / d {s’ }  PATH } ( s,  Tau( s, th^g ),  a )    d / {d th^g}  Tau( s, th^g) 
What is confusing is that they define
    A( s, a, th^p, th^g, th^v ) = sum_i   gamma^i  r_{t+1}  +  gamma^k  V(  s_{t+k}  ;  th^v  )   -   V( s_t ;  th^v )
The left side contains th^p and th^g, but the right side does not. Should these parameters be take out of the n-step advantage function A?
The second alternative for training the goal function tau seems confusing. I get that tau is going to be constrained by whatever representation PATH function was trained on and that this representation might affect the overall performance - performance.  I didn’t get the contrast with method one.  How do we treat the output of Tau as an action? Are you thinking of the gradient coming back through PATH as a reward signal? More detail here would be helpful.
EXPERIMENTS:
Lavaworld: authors show that pretraining the PATH function on longer 7-11 step policies leads to better performance
when given a specific Lava world problem to solve.  So the PATH function helps and longer paths are better. This seems reasonable. What is the upper bound on the size of PATH lengths you can train? 
Reachability: authors show that different ways of abstracting the state s into a vector encoding affect the performance of the system.  From a scientific point of view, this seems orthogonal to the point of the paper, though is relevant if you were trying to build a system. 
Taxi: the authors train the PATH problem on reachability and then show that it works for TAXI. This isn’t too surprising. Both picking up the passenger (reachability) and dropping them off somewhere are essentially the same task: moving to a point. It is interesting that the Task function is able to encode the higher level structure of the TAXI problem’s two phases.
Another task you could try is to learn to perform the same task in two different environments. Perhaps the TAXI problem, but you have two different taxis that require different actions in order to execute the same path in state space. This would require a phi(s) function that is trained in a way that doesn’t depend on the action a.
ATARI 2600 games: I am not sure what state restoration is.  Is this where you artificially return an agent to a state that would normally be hard to reach? The authors show that UA results in gains on several of the games.
The authors also demonstrate that using multiple agents with different policies can be used to collect training examples for the PATH function that improve its utility over training examples collected by a single agent policy.
RELATED WORK:
Good contrast to hierarchical learning: we don’t have switching regimes here between high-level options
I don’t understand why the authors say the PATH function can be viewed as an inverse? Oh - now I get it.
Because it takes an extended n-step transition and generates an action.