The composition of elementary behaviors to solve challenging transfer learning problems is one of the key elements in building intelligent machines. To date, there has been plenty of work on learning task-specific policies or skills but almost no focus on composing necessary, task-agnostic skills to find a solution to new problems. In this paper, we propose a novel deep reinforcement learning-based skill transfer and composition method that takes the agent's primitive policies to solve unseen tasks. We evaluate our method in difficult cases where training policy through standard reinforcement learning (RL) or even hierarchical RL is either not feasible or exhibits high sample complexity. We show that our method not only transfers skills to new problem settings but also solves the challenging environments requiring both task planning and motion control with high data efficiency.
What is the specific question/problem tackled by the paper?
This paper addresses the hierarchical RL problem of combining multiple primitive policies (pi_1, …, pi_K) into policies for more complex tasks. Given a number of primitive skills and a new task within an environment, the paper aims to learn to pick and combine the primitives as needed to solve the new task. This problem statement is interesting and the method performs well on difficult tasks. 
However, I argue for rejecting this paper because it lacks meaningful contributions to the field. I do not see how the method presented in the paper is more than RL over  hand engineered action spaces that are better for the tasks. While this improves results, we already know that for any task, there is some best action space for performing that task. This is why most HRL work aims to also find the primitive policies in additional to composing them. Is this any better than option-critic if the options are hardcoded to be the primitives? The experiment state that the option-critic method did not work, but did you give it access to the same primitives?
Summary:
The method presented in the paper is as follows: at each state s_t, the K primitives are queried for their action a_k ~ pi_k(s_t). Then, a biRNN reads in the actions in order from 1 to k. In parallel, the state and a goal are encoded by a network named “Decoder”. The encoded state and the hidden states of the RNN are used to output an attention weight over each primitive. Finally, the output action is the weighted combination of all the actions. The encoders and attention weights are trained with RL. 
This method is evaluated on several mujoco tasks, such as making a cheetah jump hurdles by combining “forward” and “jump” primitives. Each environment has predefined primitives such as “forward” “left” “right” etc. 
This method is compared against HIRO, which does not have access to the primitive policies. It is not surprising that hand engineering primitives helps performance. 
Is the approach well motivated?
The general idea behind the approach is well motivated: using primitive skills to learn complex skills is a useful goal. The details of the method are strange.
I would like to see a better motivation and empirical justification for the biRNN. Why should the primitive’s action be encode in order? The ordering of the primitives is arbitrary and constant: a fully connected network could be used, or the attentions could be output entirely independently per primitive.
In fact, I do see not why the primitives’ actions need to be encoded at all. It would be much simpler for the encoder to look at (s_t, g_t) and output a discrete probability over the K primitives.
The ablations in 5.2 are for outputting actions directly rather than mixture weights. The paper would benefit from ablations where mixture weights are output but without the biRNN or without passing in the primitive’s actions.
 
Is the method well placed in the literature? 
The main idea of predicting weights over multiple experts is not novel (see "Adaptive mixtures of local experts” from 1991). In the context of RL literature, we can interpret the primitive skills as actions directly, and then the method is performing basic RL over a better action space (the better actions being “go left”, “jump” etc. We can also interpret these as options, but unlike options a single primitive is not followed for multiple time steps with a termination condition. Functionally, this is equivalent to regular RL using domain knowledge to engineer the action space.