Generative Adversarial Networks (GANs) have proven to be a powerful framework for learning to draw samples from complex distributions. However, GANs are also notoriously difficult to train, with mode collapse and oscillations a common problem. We hypothesize that this is at least in part due to the evolution of the generator distribution and the catastrophic forgetting tendency of neural networks, which leads to the discriminator losing the ability to remember synthesized samples from previous instantiations of the generator. Recognizing this, our contributions are twofold. First, we show that GAN training makes for a more interesting and realistic benchmark for continual learning methods evaluation than some of the more canonical datasets. Second, we propose leveraging continual learning techniques to augment the discriminator, preserving its ability to recognize previous generator samples. We show that the resulting methods add only a light amount of computation, involve minimal changes to the model, and result in better overall performance on the examined image and text generation tasks.
The authors argue that catastrophic forgetting may cause mode collapse and oscillation, and propose a novel plug-and-play  regularizer that can be applied to a variety of GANs' training process to counter catastrophic forgetting of the discriminator. The regularizer is a clever adaption of EWC and IS into the context of GAN training. With the authors' formulation, this regularizer will account for the discriminator's parameter from all previous "tasks" (snapshots taken at certain iterations) with extra memory budget of only one set of parameters, while assigning higher regularization strengths to parameters learned from recent tasks. Experiments demonstrate such regularizer improves GAN models including DCGAN, SN-DCGAN, WGAN-GP on image generation tasks and textGAN on text generation tasks.
Pros:
The paper is well-written. The formulation of online memory and controlled forgetting are clever, giving rise to the adaption of EWC and IS as a practical regularizer to overcome the problem of catastrophic forgetting in GANs. The experiments also demonstrate the regularizer is superior than historical averaging and SN on the synthetic dataset, and it is able to improve multiple GAN models in both image and text generation tasks.
Cons/Suggestions:
1. Although I can see the method is working, the empirical evidence to support "mode oscillation" is not strong enough for me. I think in order for continual learning to make perfect sense, mode oscillation should be an obvious issue for GANs; otherwise, we probably don't need remembering the history, as the generator is probably evolving towards the right direction even in the vanilla approach. Still, since there have been several papers showing history is important, it should be helpful in some sense. In Figure 1, I cannot tell whether in (d), the generator returned to the previous space (probably refers to (a)). Even the centers of mass of (a) and (d) look different for me. Figure 2 (left) only shows the distribution of generated data is changing as the training proceeds in vanilla GANs, since few of them (some shallow blue lines) have low peaks in previous datasets. If the mode oscillates and the generator returns to previous state, there should at least be another peak along the line, which is missing in curves on later datasets (darker blue ones). (I guess I have understood this figure correctly, but Figure 2 seems horizontally flipped to me. Since you are testing on previous fake datasets, and the accuracy should drop on previous datasets; however, the accuracy drops on later datasets in the figure.)
2. I doubt the authors may not have tried enough sets of hyper parameters for baseline models. In table 1, the variance of GAN, GAN + l2 weight and GAN + SN are significantly higher than the others. I don't think with l2 weight regularizer, the model will be much more unstable than the authors' approach.
3. The authors didn't give the results of their regularizer with LeakGAN on text generation. Currently their model has lower test BLEU than LeakGAN, which indicates lower fluency, but its self BLEU is lower than LeakGAN, which indicates higher diversity. It would be much better if the proposed method can surpass LeakGAN on both metrics.
4. Using inception score on mixture of eight Gaussians may not make much sense, if they are using the ImageNet pre-trained model, since such a model is not trained to fit this distribution. Still, the author has reported symmetric KL. 
5. The authors did not specify their inception score on real Celeb-A and CIFAR10 images. 
Overall, I tend to accept this paper for its contribution on methods. It would be even better if my concerns could be addressed.
Edit: after seeing the review of Reviewer 3, I find the proposed method seems to be the same as Online EWC and I have downgraded the rating. The authors should address these concerns.