Event Discovery for History Representation in Reinforcement Learning
Environments in Reinforcement Learning (RL) are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about past observations. While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode. Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision. We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower. Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches.
The authors study the problem of RL under partially observed settings. While most current (D)RL approaches use RNNs to tackle this problem, RNNS are trickier to optimise than FFNNs - in practice RNN-based DRL agents can perform well on partially observed problems, may require more effort to optimise, and may underperform FFNNs on domains with no/less partial observability. The proposed solution is to use a FFNN, but provide a *history representation*, which is a set of feature vectors for previous timesteps that is extracted from a second network. The second network is trained separately using self-supervision (specifically, IIC, but adapted to use temporal consistency of observations rather than data augmentation). The proposed algorithm outperforms both PPO with a FFNN and PPO with an RNN on 5/7 Atari games (mainly games where partial observability is higher), as well as on the new and challenging Obstacle Tower benchmark - though PPO with RNN results are conspicuously missing on the latter! Given the promising approach (which also has a 2x better wall-clock training time than PPO with an RNN) and results, I would give this paper a weak accept. Some nice properties are that the instantaneous feature extractor is trained using RL, while the history feature extractor is trained using self-supervision at a high level (not pixel level), so that they are probably complementary; in addition it appears that in practice the resulting history features are sparse and usually binary, which is an intriguing and potentially useful property for future work. There are several things to be done to improve the paper however. First and foremost, the results should be run over several seeds with standard deviation/error reported (it appears that this might be the case for Obstacle Tower, but no error is reported). I believe that the large improvements in some of the domains are significant, but it would be best to have this confirmed empirically. The authors could improve the presentation of background material by giving techniques names instead of using just author names. Although it would be expensive to show how changing L affects performance on all domains, some quantitative results on this hyperparameter would be useful - the same applies to C. The authors should also provide more clarity on choosing the history head - is the head identity fixed after pretraining? It would be useful to know how the more standard PPO with RNN architecture performs, but the authors* choice of architecture is a fitting comparison to their method, and does perform well in practice on the more partially observed domains, so the current setup is satisfactory.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Event Discovery for History Representation in Reinforcement Learning
Environments in Reinforcement Learning (RL) are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about past observations. While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode. Inspired by the human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision. We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower. Using these benchmarks we show the advantage of our solution with respect to common RNN-based approaches.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors study the problem of RL under partially observed settings. While most current (D)RL approaches use RNNs to tackle this problem, RNNS are trickier to optimise than FFNNs - in practice RNN-based DRL agents can perform well on partially observed problems, may require more effort to optimise, and may underperform FFNNs on domains with no/less partial observability. The proposed solution is to use a FFNN, but provide a *history representation*, which is a set of feature vectors for previous timesteps that is extracted from a second network. The second network is trained separately using self-supervision (specifically, IIC, but adapted to use temporal consistency of observations rather than data augmentation). The proposed algorithm outperforms both PPO with a FFNN and PPO with an RNN on 5/7 Atari games (mainly games where partial observability is higher), as well as on the new and challenging Obstacle Tower benchmark - though PPO with RNN results are conspicuously missing on the latter! Given the promising approach (which also has a 2x better wall-clock training time than PPO with an RNN) and results, I would give this paper a weak accept. Some nice properties are that the instantaneous feature extractor is trained using RL, while the history feature extractor is trained using self-supervision at a high level (not pixel level), so that they are probably complementary; in addition it appears that in practice the resulting history features are sparse and usually binary, which is an intriguing and potentially useful property for future work. There are several things to be done to improve the paper however. First and foremost, the results should be run over several seeds with standard deviation/error reported (it appears that this might be the case for Obstacle Tower, but no error is reported). I believe that the large improvements in some of the domains are significant, but it would be best to have this confirmed empirically. The authors could improve the presentation of background material by giving techniques names instead of using just author names. Although it would be expensive to show how changing L affects performance on all domains, some quantitative results on this hyperparameter would be useful - the same applies to C. The authors should also provide more clarity on choosing the history head - is the head identity fixed after pretraining? It would be useful to know how the more standard PPO with RNN architecture performs, but the authors* choice of architecture is a fitting comparison to their method, and does perform well in practice on the more partially observed domains, so the current setup is satisfactory.
assistant
The authors study the problem of RL under partially observed settings. While most current (D)RL approaches use RNNs to tackle this problem, RNNS are trickier to optimise than FFNNs - in practice RNN-based DRL agents can perform well on partially observed problems, may require more effort to optimise, and may underperform FFNNs on domains with no/less partial observability. The proposed solution is to use a FFNN, but provide a *history representation*, which is a set of feature vectors for previous timesteps that is extracted from a second network. The second network is trained separately using self-supervision (specifically, IIC, but adapted to use temporal consistency of observations rather than data augmentation). The proposed algorithm outperforms both PPO with a FFNN and PPO with an RNN on 5/7 Atari games (mainly games where partial observability is higher), as well as on the new and challenging Obstacle Tower benchmark - though PPO with RNN results are conspicuously missing on the latter! Given the promising approach (which also has a 2x better wall-clock training time than PPO with an RNN) and results, I would give this paper a weak accept. Some nice properties are that the instantaneous feature extractor is trained using RL, while the history feature extractor is trained using self-supervision at a high level (not pixel level), so that they are probably complementary; in addition it appears that in practice the resulting history features are sparse and usually binary, which is an intriguing and potentially useful property for future work. There are several things to be done to improve the paper however. First and foremost, the results should be run over several seeds with standard deviation/error reported (it appears that this might be the case for Obstacle Tower, but no error is reported). I believe that the large improvements in some of the domains are significant, but it would be best to have this confirmed empirically. The authors could improve the presentation of background material by giving techniques names instead of using just author names. Although it would be expensive to show how changing L affects performance on all domains, some quantitative results on this hyperparameter would be useful - the same applies to C. The authors should also provide more clarity on choosing the history head - is the head identity fixed after pretraining? It would be useful to know how the more standard PPO with RNN architecture performs, but the authors* choice of architecture is a fitting comparison to their method, and does perform well in practice on the more partially observed domains, so the current setup is satisfactory.