The ability to learn over changing task distributions without forgetting previous knowledge, also known as continual learning, is a key enabler for scalable and trustworthy deployments of adaptive solutions. While the importance of continual learning is largely acknowledged in machine vision and reinforcement learning problems, this is mostly under-documented for sequence processing tasks. This work focuses on characterizing and quantitatively assessing the impact of catastrophic forgetting and task interference when dealing with sequential data in recurrent neural networks. We also introduce a general architecture, named Gated Incremental Memory, for augmenting recurrent models with continual learning skills, whose effectiveness is demonstrated through the benchmarks introduced in this paper.
The goal of this work is to best understand the performance and benchmarking of continual learning algorithms when applied to sequential data processing problems like language or sequence data sets. The contributions of the paper are 3 fold - new benchmarks for CL with sequential data for RNN processing, new architecture introduced for more effective processing and a thorough empirical evaluation. 
Introduction: 
I think a little more insight into why the sequential data processing CL scenario is any different than the vision scenario would be quite helpful. Specifically, it would be quite impactful to tell us more about what the additional challenges with RNNs for CL vs feedforward for CL are in the intro. 
The paper is written as if the benchmark is the main contribution and the architecture improvement is just a delta on top of this, but it gets confusing when the methods section starts off with just directly stating the new architecture. 
The algorithm seems like a straightforward combination of recurrent progressive nets and gated autoencoders for CL. Can the authors provide more justification if that is the contribution or there is more to the insight than has been previously suggested in prior work?
Figure 1 has a very uninformative caption. It also doesn’t show how modules feed into one another properly. 
The motivation for why one needs GIM after one already has A-LSTM or A-LMN is not very clear?
Overall the contribution does seem a bit incremental based on prior work and the description lacks enough detail to properly indicate why this is a very important contribution?
Experiments:
What does it mean to be application agnostic but restricted to particular datasets and losses? This doesn’t quite parse to me. 
The description of the tasks is very informal and hard to follow. It’s not clear what exactly the tasks and datasets look like 
“using morehidden units can bridge this gap” -> why not just do it? Its a benchmark after all. 
Overall the task descriptions should be in a separate section where the setup is described in a lot of detail and motivated properly. 
The results in the experiments section are very hard to parse. The captions need much more detail for eg Table 2. 
Could we also possibly have more baselines from continual learning? For instance EWC (Kirkpatrick) or generative replay might be competitive baselines. 
Overall I think that the GIM and A-LMN and A-LSTM methods are reasonable although somewhat incremental. But the proposed benchmarks are pretty unclear and the results are a bit hard to really interpret well. It would also be important to run comparisons with more baselines and to provide more ablation/analysis experiments to really see the benefit of GIM/A-LMN or A-LSTM. I also think that the task descriptions should be much earlier in the paper and desribed in much more rigorous detail.