Environments in Reinforcement Learning (RL) are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about past observations. While common methods represent this history using a Recurrent Neural Network (RNN), in this paper we propose an alternative representation which is based on the record of the past events observed in a given episode. Inspired by human memory, these events describe only important changes in the environment and, in our approach, are automatically discovered using self-supervision. We evaluate our history representation method using two challenging RL benchmarks: some games of the Atari-57 suite and the 3D environment Obstacle Tower. Using these benchmarks we show the advantage of our solution with respect to common approaches.
The paper focuses on the problem learning state representations that can effectively capture historical information
in POMDPs. Specifically, the paper proposes an alternative approach to using RNN's for capturing such history - the authors adopt a variant of recently proposed Invariant Information Clustering approach to discover important events in past observation and then use these events to learn a probability distribution over observations to represent the state information. The goal here is to address the unstable and inefficient training issues associated with RNN based architectures. The authors validate their approach with experiments on seven tasks on Atari 57 benchmark and Obstacle Tower, both with discrete action space and compare their performance against both original and RNN versions of PPO. 
The paper addresses an interesting problem to help learning better state representation in absence of  complete information about the environment. The approach of using IIC clustering (or any clustering approach) for learning state representations is novel. The overall goal of replacing RNN based methods in order to achieve stable, efficient learning in presence of budget constraints is very useful and hence this approach is potentially a good step in that direction. Although not adequate, the results in figure 3 provides good insight into effectiveness of the method in making training easier. 
However, I am inclining to reject this paper for the following reasons:
(1) The motivation for using proposed clustering approach for history representation is neither clear  and nor well exposed. MI based methods are inherently difficult to learn and hence this approach needs  rigorous analysis on why it works when it does and how it fails.
(2) The paper fails to position the new approach in comparison to related works both in discussions and experiments.
(3) The experiments are very limited in nature and fails to demonstrate the efficacy of the proposed approach effectively. Further, the key contribution focusing on learning effective representations under constrained budget is not adequately tested.
Major concerns:
Motivation
-------------
- It is not clear how the proposed clustering mechanism to discover events allows to successfully capture information that an RNN based approach does. For instance, RNN helps to capture long-term dependencies but it is very hard to interpret the proposed model from that aspect. Also, why is this particular clustering approach (Invariant Information Clustering) chosen? The motivation for using this approach is not clear and overall combination appears adhoc.
- Further, RNN based methods can retain order information in sequential history of observations. However,  this method appears to not consider that information. Is this is true or am I mistaken here? If this is true, why would this not create issues in learning good representations for task where order is indeed important? 
- The paper discusses several articles that provide background on the methods used however it fails to position the exposition in comparison to existing RNN based approaches which is a big miss as the goal of the paper is to replace RNN based approaches [1,2,3,4].
Method
------
- In the event discovery stage, it is not clear why using consecutive observations is not useful. Also, if one does use L=1 (consecutive observations), can one reduce this method to RNN as you end of capturing all the previous history? Also, why L=3 is good across all different tasks? does it have any relation with the use of 4 frames in I(t)? 
- The authors mention tat H(t) matrix is highly sparse but also low-dimensional in all their experiments.  Would this is be the case for any other task? If not, is this a limitation of the method that you need S and C  to be small? 
- The authors attempt to use clustering based approach with the hope of recovering important information, however
mention that H(t) stores all the past events. Isn't this contradictory? Also, why can RNN with an attention based mechanism not achieve similar effect?
- It is also useful to analyse how will this method work in presence of long vs short history? Will the clustering itself and hence the learned representations get affected by length of available history?
Experiments
----------
- Focusing only on PPO and designing an RNN version of PPO constrains the effectiveness of experiments in validating the approach. Authors mention difficulty of training with RNN as one reason for PPO with RNN's under performance but this is not convincing Could the performance of PPO with RNN be limited only due to specific RNN architecture used?
- Authors must compare with other methods that use RNN approaches (e.g. [1]). Also, if methods such [2],[3],[4] are not  directly applicable, they must atleast, use their RNN based architectures to modify PPO and compare several baselines.
- Why do the authors not report PPO with RNN for Obstacle Tower? 
- For the experiments, authors use a specific 10000 steps budget but this seems to be highly curated. The experiments
would be stronger if the authors show experiment over a range of budget. This will also give insights on when does RNN becomes better and is there a budget after which both methods perform equally well or RNN based approaches
surpass the current approaches.
- Figure 3: Training dynamics seem to favor RNN approach for BreakOut, Gravitar. Do the authors have insight on why
this is the case?
 
Minor points to improve submission not affecting the score:
- The paper needs to be proofread for various typos and sentence construction issues
- Table 1: For Qbert, original PPO (Sch.) is best performing, not EDHR
- [5] talks about difficulties in MI based methods and I encourage the authors to look at the analysis in
this paper. As it is only an arXiv version and not published yet, I have not based my assessment on the
existence of this paper but still connection to such analysis will make this paper stronger.
[1] Deep Variational Reinforcement Learning for POMDPs, Igl et. al.
[2] On improving deep reinforcement learning for POMDPs, Zhu et. al.
[3] Policy Learning with continuous memory states in partially observed robotic control, Zhang et. al.
[4] Memory-based control with recurrent neural networks, Heess et. al. 
[5] On Mutual Information Maximization and Representation Learning, Tschannen et. al.