Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training. Various selective synaptic plasticity approaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks. We explore such selective synaptic plasticity approaches through a unifying lens of memory replay and show the close relationship between methods like Elastic Weight Consolidation (EWC) and Memory-Aware-Synapses (MAS). We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network's output at an arbitrary layer for previous tasks while learning a new one. We propose the sliced Cramér distance as a suitable choice for such preservation and evaluate our Sliced Cramér Preservation (SCP) algorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings. We show that SCP consistently utilizes the learning capacity of the network better than online-EWC and MAS methods on various incremental learning tasks.
[Summary]
This paper proposes a new method for overcoming catastrophic forgetting in continual learning, based on distribution-based regularization using the sliced Cramer distance, i.e. Sliced Cramer Preservation (SCP). Unlike previous work on catastrophic forgetting, this paper tackles unsupervised learning scenarios as well as supervised learning. They evaluate the proposed SCP on permutated MNIST, sequential learning in autoencoder task, and sequential learning for segmentation. 
[Pros]
- This paper tackles unsupervised learning scenarios beyond the classification on benchmark datasets.
- This paper employes sliced Cramer distance with theoretical justification.
- The analysis on EwC and MAS in terms of geometric view
- Experimental results look promising.
[Cons]  
- Even if MAS[1] describes the details on synaptic concept and Hebbian rule, many readers might be not familiar with synaptic or neuro-science terms. So, in prelimiary session, more explanation can help readers to understand.
- In addition to permute-MNIST, it is required to be evaluated on more conventional tasks such as MNIST->SVHN or CIFAR-10, 100 datasets. Also, more recent work should be compared such as IMM [2] and PGMA [3] for supervised learning scenarios.
- All graph result figures can be improved for enhancing legibility. In Figure 6, In particular, the subtitle of Summer case confuses me.
 
[1] Aljundi et al. Memory Aware Synapses: Learning what (not) to forget, ECCV 2018.
[2] Lee et al. Overcoming Catastrophic Forgetting by Incremental Moment Matching, NIPS 2017.
[3] Hu et al. Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation, ICLR 2019.