RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning
Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a “fast” reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL^2, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose (“slow”) RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the “fast” RL algorithm on the current (previously unseen) MDP. We evaluate RL^2 experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL^2 is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL^2 on a vision-based navigation task and show that it scales up to high-dimensional problems.
The authors try to address the issue of data efficiency in deep reinforcement learning by meta-learning a reinforcement learning algorithm using a hand-designed reinforcement learning algorithm (TRPO in this case). The experiments suggest comparable performance to models with prior knowledge of the distribution over environments for bandit tasks, and experiments on random maze navigation from vision is shown as well, though the random maze experiments would benefit from a clearer explanation. It was not obvious from the text how their experiments supported the thesis of the paper that the learned RL algorithm was effectively performing one-shot learning. The subject of the paper is also strikingly similar to the recently-posted paper Learning to Reinforcement Learn (https://arxiv.org/pdf/1611.05763.pdf), and while this paper was posted after the ICLR deadline, the authors should probably update the text to reflect the state of this rapidly-advancing field.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning
Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a “fast” reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL^2, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose (“slow”) RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the “fast” RL algorithm on the current (previously unseen) MDP. We evaluate RL^2 experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL^2 is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL^2 on a vision-based navigation task and show that it scales up to high-dimensional problems.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors try to address the issue of data efficiency in deep reinforcement learning by meta-learning a reinforcement learning algorithm using a hand-designed reinforcement learning algorithm (TRPO in this case). The experiments suggest comparable performance to models with prior knowledge of the distribution over environments for bandit tasks, and experiments on random maze navigation from vision is shown as well, though the random maze experiments would benefit from a clearer explanation. It was not obvious from the text how their experiments supported the thesis of the paper that the learned RL algorithm was effectively performing one-shot learning. The subject of the paper is also strikingly similar to the recently-posted paper Learning to Reinforcement Learn (https://arxiv.org/pdf/1611.05763.pdf), and while this paper was posted after the ICLR deadline, the authors should probably update the text to reflect the state of this rapidly-advancing field.
assistant
The authors try to address the issue of data efficiency in deep reinforcement learning by meta-learning a reinforcement learning algorithm using a hand-designed reinforcement learning algorithm (TRPO in this case). The experiments suggest comparable performance to models with prior knowledge of the distribution over environments for bandit tasks, and experiments on random maze navigation from vision is shown as well, though the random maze experiments would benefit from a clearer explanation. It was not obvious from the text how their experiments supported the thesis of the paper that the learned RL algorithm was effectively performing one-shot learning. The subject of the paper is also strikingly similar to the recently-posted paper Learning to Reinforcement Learn (https://arxiv.org/pdf/1611.05763.pdf), and while this paper was posted after the ICLR deadline, the authors should probably update the text to reflect the state of this rapidly-advancing field.