Multi-task learning with deep model based reinforcement learning
In recent years, model-free methods that use deep learning have achieved great success in many different reinforcement learning environments. Most successful approaches focus on solving a single task, while multi-task reinforcement learning remains an open problem. In this paper, we present a model based approach to deep reinforcement learning which we use to solve different tasks simultaneously. We show that our approach not only does not degrade but actually benefits from learning multiple tasks. For our model, we also present a new kind of recurrent neural network inspired by residual networks that decouples memory from computation allowing to model complex environments that do not require lots of memory. The code will be released before ICLR 2017.
This paper proposes a model-based reinforcement learning approach focusing on predicting future rewards given a current state and future actions. This is achieved with a *residual recurrent neural network*, that outputs the expected reward increase at various time steps in the future. To demonstrate the usefulness of this approach, experiments are conducted on Atari games, with a simple playing strategy that consists in evaluating random sequences of moves and picking the one with highest expected reward (and low enough chance of dying). Interestingly, out of the 3 games tested, one of them exhibits better performance when the agent is trained in a multitask setting (i.e. learning all games simultaneously), hinting that transfer learning is occurring. This submission is easy enough to read, and the reward prediction architecture looks like an original and sound idea. There are however several points that I believe prevent this work from reaching the ICLR bar, as detailed below. The first issue is the discrepancy between the algorithm proposed in Section 3 vs its actual implementation in Section 4 (experiments): in Section 3 the output is supposed to be the expected accumulated reward in future time steps (as a single scalar), while in experiments it is instead two numbers, one which is the probability of dying and another one which is the probability of having a higher score without dying. This might work better, but it also means the idea as presented in the main body of the paper is not actually evaluated (and I guess it would not work well, as otherwise why implement it differently?) In addition, the experimental results are quite limited: only on 3 games that were hand-picked to be easy enough, and no comparison to other RL techniques (DQN & friends). I realize that the main focus of the paper is not about exhibiting state-of-the-art results, since the policy being used is only a simple heuristic to show that the model predictions can ne used to drive decisions. That being said, I think experiments should have tried to demonstrate how to use this model to obtain better reinforcement learning algorithms: there is actually no reinforcement learning done here, since the model is a supervised algorithm, used in a manually-defined hardcoded policy. Another question that could have been addressed (but was not) in the experiments is how good these predictions are (e.g. classification error on dying probability, MSE on future rewards, ...), compared to simpler baselines. Finally, the paper*s *previous work* section is too limited, focusing only on DQN and in particular saying very little on the topic of model-based RL. I think a paper like for instance *Action-Conditional Video Prediction using Deep Networks in Atari Games* should have been an obvious *must cite*. Minor comments: - Notations are unusual, with *a* denoting a state rather than an action, this is potentially confusing and I see no reason to stray away from standard RL notations - Using a dot for tensor concatenation is not a great choice either, since the dot usually indicates a dot product - The r_i in 3.2.2 is a residual that has nothing to do with r_i the reward - c_i is defined as *The control that was performed at time i*, but instead it seems to be the control performed at time i-1 - There is a recurrent confusion between mean and median in 3.2.2 - x should not be used in Observation 1 since the x from Fig. 3 does not go through layer normalization - The inequality in Observation 1 should be about |x_i|, not x_i - Observation 1 (with its proof) takes too much space for such a simple result - In 3.2.3 the first r_j should be r_i - The probability of dying comes out of nowhere in 3.3, since we do not know yet it will be an output of the model - *Our approach is not able to learn from good strategies* => did you mean **only* from good strategies*? - Please say that in Fig. 4 *fc* means *fully connected* - It would be nice also to say how the architecture of Fig. 4 differs from the classical DQN architecture from Mnih et al (2015) - Please clarify r_j2 as per your answer in OpenReview comments - Table 3 says *After one iteration* but has *PRL Iteration 2* in it, which is confusing - *Figure 5 shows that not only there is no degradation in Pong and Demon Attack*=> to me it seems to be a bit worse, actually - *A model that has learned only from random play is able to play at least 7 times better.* => not clear where this 7 comes from - *Demon Attack*s plot in Figure 5c shows a potential problem we mentioned earlier* => where was it mentioned?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Multi-task learning with deep model based reinforcement learning
In recent years, model-free methods that use deep learning have achieved great success in many different reinforcement learning environments. Most successful approaches focus on solving a single task, while multi-task reinforcement learning remains an open problem. In this paper, we present a model based approach to deep reinforcement learning which we use to solve different tasks simultaneously. We show that our approach not only does not degrade but actually benefits from learning multiple tasks. For our model, we also present a new kind of recurrent neural network inspired by residual networks that decouples memory from computation allowing to model complex environments that do not require lots of memory. The code will be released before ICLR 2017.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes a model-based reinforcement learning approach focusing on predicting future rewards given a current state and future actions. This is achieved with a *residual recurrent neural network*, that outputs the expected reward increase at various time steps in the future. To demonstrate the usefulness of this approach, experiments are conducted on Atari games, with a simple playing strategy that consists in evaluating random sequences of moves and picking the one with highest expected reward (and low enough chance of dying). Interestingly, out of the 3 games tested, one of them exhibits better performance when the agent is trained in a multitask setting (i.e. learning all games simultaneously), hinting that transfer learning is occurring. This submission is easy enough to read, and the reward prediction architecture looks like an original and sound idea. There are however several points that I believe prevent this work from reaching the ICLR bar, as detailed below. The first issue is the discrepancy between the algorithm proposed in Section 3 vs its actual implementation in Section 4 (experiments): in Section 3 the output is supposed to be the expected accumulated reward in future time steps (as a single scalar), while in experiments it is instead two numbers, one which is the probability of dying and another one which is the probability of having a higher score without dying. This might work better, but it also means the idea as presented in the main body of the paper is not actually evaluated (and I guess it would not work well, as otherwise why implement it differently?) In addition, the experimental results are quite limited: only on 3 games that were hand-picked to be easy enough, and no comparison to other RL techniques (DQN & friends). I realize that the main focus of the paper is not about exhibiting state-of-the-art results, since the policy being used is only a simple heuristic to show that the model predictions can ne used to drive decisions. That being said, I think experiments should have tried to demonstrate how to use this model to obtain better reinforcement learning algorithms: there is actually no reinforcement learning done here, since the model is a supervised algorithm, used in a manually-defined hardcoded policy. Another question that could have been addressed (but was not) in the experiments is how good these predictions are (e.g. classification error on dying probability, MSE on future rewards, ...), compared to simpler baselines. Finally, the paper*s *previous work* section is too limited, focusing only on DQN and in particular saying very little on the topic of model-based RL. I think a paper like for instance *Action-Conditional Video Prediction using Deep Networks in Atari Games* should have been an obvious *must cite*. Minor comments: - Notations are unusual, with *a* denoting a state rather than an action, this is potentially confusing and I see no reason to stray away from standard RL notations - Using a dot for tensor concatenation is not a great choice either, since the dot usually indicates a dot product - The r_i in 3.2.2 is a residual that has nothing to do with r_i the reward - c_i is defined as *The control that was performed at time i*, but instead it seems to be the control performed at time i-1 - There is a recurrent confusion between mean and median in 3.2.2 - x should not be used in Observation 1 since the x from Fig. 3 does not go through layer normalization - The inequality in Observation 1 should be about |x_i|, not x_i - Observation 1 (with its proof) takes too much space for such a simple result - In 3.2.3 the first r_j should be r_i - The probability of dying comes out of nowhere in 3.3, since we do not know yet it will be an output of the model - *Our approach is not able to learn from good strategies* => did you mean **only* from good strategies*? - Please say that in Fig. 4 *fc* means *fully connected* - It would be nice also to say how the architecture of Fig. 4 differs from the classical DQN architecture from Mnih et al (2015) - Please clarify r_j2 as per your answer in OpenReview comments - Table 3 says *After one iteration* but has *PRL Iteration 2* in it, which is confusing - *Figure 5 shows that not only there is no degradation in Pong and Demon Attack*=> to me it seems to be a bit worse, actually - *A model that has learned only from random play is able to play at least 7 times better.* => not clear where this 7 comes from - *Demon Attack*s plot in Figure 5c shows a potential problem we mentioned earlier* => where was it mentioned?
assistant
This paper proposes a model-based reinforcement learning approach focusing on predicting future rewards given a current state and future actions. This is achieved with a *residual recurrent neural network*, that outputs the expected reward increase at various time steps in the future. To demonstrate the usefulness of this approach, experiments are conducted on Atari games, with a simple playing strategy that consists in evaluating random sequences of moves and picking the one with highest expected reward (and low enough chance of dying). Interestingly, out of the 3 games tested, one of them exhibits better performance when the agent is trained in a multitask setting (i.e. learning all games simultaneously), hinting that transfer learning is occurring. This submission is easy enough to read, and the reward prediction architecture looks like an original and sound idea. There are however several points that I believe prevent this work from reaching the ICLR bar, as detailed below. The first issue is the discrepancy between the algorithm proposed in Section 3 vs its actual implementation in Section 4 (experiments): in Section 3 the output is supposed to be the expected accumulated reward in future time steps (as a single scalar), while in experiments it is instead two numbers, one which is the probability of dying and another one which is the probability of having a higher score without dying. This might work better, but it also means the idea as presented in the main body of the paper is not actually evaluated (and I guess it would not work well, as otherwise why implement it differently?) In addition, the experimental results are quite limited: only on 3 games that were hand-picked to be easy enough, and no comparison to other RL techniques (DQN & friends). I realize that the main focus of the paper is not about exhibiting state-of-the-art results, since the policy being used is only a simple heuristic to show that the model predictions can ne used to drive decisions. That being said, I think experiments should have tried to demonstrate how to use this model to obtain better reinforcement learning algorithms: there is actually no reinforcement learning done here, since the model is a supervised algorithm, used in a manually-defined hardcoded policy. Another question that could have been addressed (but was not) in the experiments is how good these predictions are (e.g. classification error on dying probability, MSE on future rewards, ...), compared to simpler baselines. Finally, the paper*s *previous work* section is too limited, focusing only on DQN and in particular saying very little on the topic of model-based RL. I think a paper like for instance *Action-Conditional Video Prediction using Deep Networks in Atari Games* should have been an obvious *must cite*. Minor comments: - Notations are unusual, with *a* denoting a state rather than an action, this is potentially confusing and I see no reason to stray away from standard RL notations - Using a dot for tensor concatenation is not a great choice either, since the dot usually indicates a dot product - The r_i in 3.2.2 is a residual that has nothing to do with r_i the reward - c_i is defined as *The control that was performed at time i*, but instead it seems to be the control performed at time i-1 - There is a recurrent confusion between mean and median in 3.2.2 - x should not be used in Observation 1 since the x from Fig. 3 does not go through layer normalization - The inequality in Observation 1 should be about |x_i|, not x_i - Observation 1 (with its proof) takes too much space for such a simple result - In 3.2.3 the first r_j should be r_i - The probability of dying comes out of nowhere in 3.3, since we do not know yet it will be an output of the model - *Our approach is not able to learn from good strategies* => did you mean **only* from good strategies*? - Please say that in Fig. 4 *fc* means *fully connected* - It would be nice also to say how the architecture of Fig. 4 differs from the classical DQN architecture from Mnih et al (2015) - Please clarify r_j2 as per your answer in OpenReview comments - Table 3 says *After one iteration* but has *PRL Iteration 2* in it, which is confusing - *Figure 5 shows that not only there is no degradation in Pong and Demon Attack*=> to me it seems to be a bit worse, actually - *A model that has learned only from random play is able to play at least 7 times better. * => not clear where this 7 comes from - *Demon Attack*s plot in Figure 5c shows a potential problem we mentioned earlier* => where was it mentioned?