Deep reinforcement learning (RL) algorithms have made great strides in recent years. An important remaining challenge is the ability to quickly transfer existing skills to novel tasks, and to combine existing skills with newly acquired ones. In domains where tasks are solved by composing skills this capacity holds the promise of dramatically reducing the data requirements of deep RL algorithms, and hence increasing their applicability. Recent work has studied ways of composing behaviors represented in the form of action-value functions. We analyze these methods to highlight their strengths and weaknesses, and point out situations where each of them is susceptible to poor performance. To perform this analysis we extend generalized policy improvement to the max-entropy framework and introduce a method for the practical implementation of successor features in continuous action spaces. Then we propose a novel approach which, in principle, recovers the optimal policy during transfer. This method works by explicitly learning the (discounted, future) divergence between policies. We study this approach in the tabular case and propose a scalable variant that is applicable in multi-dimensional continuous action spaces. We compare our approach with existing ones on a range of non-trivial continuous control problems with compositional structure, and demonstrate qualitatively better performance despite not requiring simultaneous observation of all task rewards.
The authors introduce Divergence Correction (DC) for the problem of transfer learning by composing policies. There approach builds on GPI with a maximum entropy objective. They also prove that DC solves for the max-entropy optimal interpolation between two policies and derive a practical approximation for this algorithm. They provide experimental results in a gridworld problem and study their approximate algorithm in two continuous control problems.
While this paper has some interesting ideas (combining GPI with a Max-Entropy objective and DC), these ideas are not properly motivated. The main problem seems to be clarity. One big problem is that the paper never defines the notion of a notion of optimality (or near-optimality). Also, considering that the DC algorithm is one of the main contributions of the paper it is barely motivated. Theorem 3.2 is presented with almost no explanation about how DC was derived. Why do the authors believe that DC is a good idea on a conceptual level? It's very interesting that the paper presents cases where previous approaches (Optimistic and GPI) don't perform well. But the authors don't explain why they believe DC should perform well in these cases. 
The authors make the unjustified claim in the abstract that their approach has "near-optimal performance and requires less information". I say this is unjustified because they only try this approach on three benchmarks. In addition, there should be situations where DC also performs poorly since there are known hardness results for solving MDPs. Admittedly, those results may not apply if the authors are making assumptions that are not being clearly discussed in the paper.
Minor Comments:
1. In the abstract, "requiring less information" is very imprecise. Are you referring to sample complexity?
2. In the introduction, "can consistently achieve good performance" is imprecise. What is the notion of near-optimality? What does consistent mean? Having experimental results on 3 tasks doesn't seem to be enough to me to justify this claim.
3. In the introduction (and rest of the paper), please don't call Haarnoja et al.'s approach optimistic. Optimism already has another widely used meaning in RL literature. Maybe call it "Uncorrected".
4. In section 2.2, the authors introduce \pi_1, \pi_2, ... , \pi_n but never actually use that notation. This section does not clearly explain how GPI works.
5. In Theorem 3.1, the authors should introduce Q^1, Q^2, ... , Q^n and define the policies in terms of the action-value functions. Also, the statement of this theorem is not self contained, what is the reward function of the MDP? The proof below should be called a proof sketch.
6. The paper mentions that extending to multiple tasks is possible. Is it trivial? What is the basic idea? It seems straightforward but it might be helpful to explicitly state the idea.
7. In Theorem 3.2, how was C derived? Please add some commentary explaining the conceptual idea.
8. In Table 1, what is f(s, a|b)? I don't see where this was defined?
9. CondQ is usually referred to as UVFA in the literature.
10. Section 3 really needs a conclusion statement.
11. Section 4 is very unclear and hard to follow.
12. In figure 1f, what is LTD? It's never defined. I'm guessing it's DC.
13. All of the figures are too small and some are not clear in black and white.