Current generation of memory-augmented neural networks has limited scalability as they cannot efficiently process data that are too large to fit in the external memory storage. One example of this is lifelong learning scenario where the model receives unlimited length of data stream as an input which contains vast majority of uninformative entries. We tackle this problem by proposing a memory network fit for long-term lifelong learning scenario, which we refer to as Long-term Episodic Memory Networks (LEMN), that features a RNN-based retention agent that learns to replace less important memory entries based on the retention probability generated on each entry that is learned to identify data instances of generic importance relative to other memory entries, as well as its historical importance. Such learning of retention agent allows our long-term episodic memory network to retain memory entries of generic importance for a given task. We validate our model on a path-finding task as well as synthetic and real question answering tasks, on which our model achieves significant improvements over the memoryaugmented networks with rule-based memory scheduling as well as an RL-based baseline that does not consider relative or historical importance of the memory.
This work tackles the problems encountered by bounded memory storage mechanisms when faced with abundant data, of which much may be irrelevant or redundant. Such a problem is faced in lifelong learning settings, where a limitless data stream must somehow be encoded and stored so as to be useful at later points in time. 
The researchers propose a solution based on “learning what to remember”. That is, rather than encode every observation (which can quickly become problematic), the model learns to replace less important memories. The importance of a memory is determined by its correlation with future reward; a “memory retention policy” is learned via reinforcement learning, wherein the model learns to retain or discard memories based on these actions’ (i.e., retentions) impact on future reward. Experiments to show the effectiveness of this mechanism include gridworld IMaze and Random Mazes, bAbI question answering (task 2), and Trivia QA. 
Altogether the work does well to clearly describe an interesting approach to an important problem. The model is motivated and explained well, and there were no issues with understanding its inner workings. 
Regarding the work’s novelty, there is a precedent for using RL-based write schemes (DNTM from Gulcehre et al, 2016), which the authors point out. I am not entirely convinced that the proposed writing scheme is a substantial addition over this past work, but I am not overly concerned about this since proper due credit is assigned in the paper. Perhaps a bit more discussion about the advantages of the proposed writing scheme could go a long way, since as it stands now, the paper simply claims that this past work “only considers the pairwise relationships between the current data instance and each individual memory”, and I’m not sure how much substance actually underlies this difference.
Unfortunately I think there is a fundamental problem with the work. The model is a proposed solution for problems with vast amounts of streaming data; problems that, presumably, current memory models would struggle with. However, the tasks in the paper do not fall in this domain. Instead, the authors chose to artificially cripple the size of their memory (using, for example, just a handful of memory “slots”) and demonstrate its performance on tasks that are otherwise completely within the realm of being solved by conventional memory models. This is fine as a jumping off point for the research, but for the model to be taken seriously as a valid solution to problems involving such a scale of data that current models cannot even cope, then it needs to show its worth on problems involving such a scale of data that current models cannot cope. 
Demonstrating success here is important for a few reasons. First, such high-data scenarios may involve situations where many, many memories need to be encoded and considered for the future, since they are all useful or necessary for future performance. The experiments do not show whether the model can scale to, say, 100 or 1000 memories, which is within the realm of being “reasonable” for current memory architectures. Second, high-data scenarios may involve an abundant amount of distracting, irrelevant data. This places particularly tough demands on the RL-based writing mechanism, which will undoubtedly face problems with temporal credit assignment if: (a) the time between encoding and retrieval is long, and (b) there is high reward noise in the intermediate time. Thus, the authors should stress-test the components of their model, since these stresses will undoubtedly exist in the problems that the model is proposed to solve.
Some other minor considerations include the following. (1) The use of a single bAbI task is questionable. Why not run the model on the full suite? (2) How do conventional memory models perform on the tasks? Why are the baselines only variants of the proposed model? 
To conclude and summarize, as a proposed solution to scenarios with streams of abundant data -- which the authors claim is a domain that current memory models may struggle -- the proposed model should tackle problems that: 1) have characteristics more reminiscent of these scenarios, and 2) are problems on which current memory models struggle, for the reasons claimed in the paper. In particular, it would be valuable to see model performance on tasks wherein very long stretches of time need to be considered. This is important because it can address questions with memory scaling (how does the model cope with more than a handful of memories?), and issues that would crop up in a reinforcement learning-based approach to memory retention over long time intervals (namely, long-term temporal credit assignment).