Disentangled Cumulants Help Successor Representations Transfer to New Tasks
Biological intelligence can learn to solve many diverse tasks in a data efficient manner by re-using basic knowledge and skills from one task to another. Furthermore, many of such skills are acquired through something called latent learning, where no explicit supervision for skill acquisition is provided. This is in contrast to the state-of-the-art reinforcement learning agents, which typically start learning each new task from scratch and struggle with knowledge transfer. In this paper we propose a principled way to learn and recombine a basis set of policies, which comes with certain guarantees on the coverage of the final task space. In particular, we construct a learning pipeline where an agent invests time to learn to perform intrinsically generated, goal-based tasks, and subsequently leverages this experience to quickly achieve a high level of performance on externally specified, often significantly more complex tasks through generalised policy improvement. We demonstrate both theoretically and empirically that such goal-based intrinsic tasks produce more transferable policies when the goals are specified in a space that exhibits a form of disentanglement.
The paper tackles the challenging problem of transfer learning and few shot learning in RL setting and provides some theoretical guarantees for the downstream task coverage. The paper structure can be further improved by adding a background subsection on successor representation (SR) in RL; SR is not a very well known representation in RL and a brief subsection on that can help the reader in understanding the motivation behind using it. In terms of related work ,another work which can also be mentioned (although not directly related) is “DARLA: Improving Zero-Shot Transfer in Reinforcement Learning” which also uses disentangled representations for zero-shot transfer learning. The paper also needs to be more clear in terms of contributions; it seems that there is a significant overlap between this work and (Barreto et al., 2017, 2018); some clarification would be helpful here. In terms of empirical results; the authors can also compare with other transfer learning methods in deep RL such as Hansen 2019 or Nair 2018 or explain why these are not reasonable baselines. Also the results for DIAYN are a bit surprising to me since in all the experiments the performance of the method is underwhelming; this is especially surprising because in the original DIAYN paper the method performed well in reasonably complex tasks. Can you provide an intuition on why DIAYN performs poorly even in the agent tasks.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Disentangled Cumulants Help Successor Representations Transfer to New Tasks
Biological intelligence can learn to solve many diverse tasks in a data efficient manner by re-using basic knowledge and skills from one task to another. Furthermore, many of such skills are acquired through something called latent learning, where no explicit supervision for skill acquisition is provided. This is in contrast to the state-of-the-art reinforcement learning agents, which typically start learning each new task from scratch and struggle with knowledge transfer. In this paper we propose a principled way to learn and recombine a basis set of policies, which comes with certain guarantees on the coverage of the final task space. In particular, we construct a learning pipeline where an agent invests time to learn to perform intrinsically generated, goal-based tasks, and subsequently leverages this experience to quickly achieve a high level of performance on externally specified, often significantly more complex tasks through generalised policy improvement. We demonstrate both theoretically and empirically that such goal-based intrinsic tasks produce more transferable policies when the goals are specified in a space that exhibits a form of disentanglement.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper tackles the challenging problem of transfer learning and few shot learning in RL setting and provides some theoretical guarantees for the downstream task coverage. The paper structure can be further improved by adding a background subsection on successor representation (SR) in RL; SR is not a very well known representation in RL and a brief subsection on that can help the reader in understanding the motivation behind using it. In terms of related work ,another work which can also be mentioned (although not directly related) is “DARLA: Improving Zero-Shot Transfer in Reinforcement Learning” which also uses disentangled representations for zero-shot transfer learning. The paper also needs to be more clear in terms of contributions; it seems that there is a significant overlap between this work and (Barreto et al., 2017, 2018); some clarification would be helpful here. In terms of empirical results; the authors can also compare with other transfer learning methods in deep RL such as Hansen 2019 or Nair 2018 or explain why these are not reasonable baselines. Also the results for DIAYN are a bit surprising to me since in all the experiments the performance of the method is underwhelming; this is especially surprising because in the original DIAYN paper the method performed well in reasonably complex tasks. Can you provide an intuition on why DIAYN performs poorly even in the agent tasks.
assistant
The paper tackles the challenging problem of transfer learning and few shot learning in RL setting and provides some theoretical guarantees for the downstream task coverage. The paper structure can be further improved by adding a background subsection on successor representation (SR) in RL; SR is not a very well known representation in RL and a brief subsection on that can help the reader in understanding the motivation behind using it. In terms of related work ,another work which can also be mentioned (although not directly related) is “DARLA: Improving Zero-Shot Transfer in Reinforcement Learning” which also uses disentangled representations for zero-shot transfer learning. The paper also needs to be more clear in terms of contributions; it seems that there is a significant overlap between this work and (Barreto et al., 2017, 2018); some clarification would be helpful here. In terms of empirical results; the authors can also compare with other transfer learning methods in deep RL such as Hansen 2019 or Nair 2018 or explain why these are not reasonable baselines. Also the results for DIAYN are a bit surprising to me since in all the experiments the performance of the method is underwhelming; this is especially surprising because in the original DIAYN paper the method performed well in reasonably complex tasks. Can you provide an intuition on why DIAYN performs poorly even in the agent tasks.