Published as a conference paper at ICLR 2018 MEMORIZATION PRECEDES GENERATION: LEARNING UNSUPERVISED GANS WITH MEMORY NETWORKS
We propose an approach to address two issues that commonly occur during training of unsupervised GANs. First, since GANs use only a continuous latent distribution to embed multiple classes or clusters of data, they often do not correctly handle the structural discontinuity between disparate classes in a latent space. Second, discriminators of GANs easily forget about past generated samples by generators, incurring instability during adversarial training. We argue that these two infamous problems of unsupervised GAN training can be largely alleviated by a learnable memory network to which both generators and discriminators can access. Generators can effectively learn representation of training samples to understand underlying cluster distributions of data, which ease the structure discontinuity problem. At the same time, discriminators can better memorize clusters of previously generated samples, which mitigate the forgetting problem. We propose a novel end-to-end GAN model named memoryGAN, which involves a memory network that is unsupervisedly trainable and integrable to many existing GAN models. With evaluations on multiple datasets such as Fashion-MNIST, CelebA, CIFAR10, and Chairs, we show that our model is probabilistically interpretable, and generates realistic image samples of high visual fidelity. The memoryGAN also achieves the state-of-the-art inception scores over unsupervised GAN models on the CIFAR10 dataset, without any optimization tricks and weaker divergences.
[Overview]
In this paper, the authors proposed a novel model called MemoryGAN, which integrates memory network with GAN. As claimed by the authors, MemoryGAN is aimed at addressing two problems of GAN training: 1) difficult to model the structural discontinuity between disparate classes in the latent space; 2) catastrophic forgetting problem during the training of discriminator about the past synthesized samples by the generator. It exploits the life-long memory network and adapts it to GAN. It consists of two parts, discriminative memory network (DMN) and Memory Conditional Generative Network (MCGN). DMN is used for discriminating input samples by integrating the memory learnt in the memory network, and MCGN is used for generating images based on random vector and the sampled memory from the memory network. In the experiments, the authors evaluated memoryGAN on three datasets, CIFAR-10, affine-MNIST and Fashion-MNIST, and demonstrated the superiority to previous models. Through ablation study, the authors further showed the effects of separate components in memoryGAN. 
[Strengths]
1. This paper is well-written. All modules in the proposed model and the experiments were explained clearly. I enjoyed much to read the paper.
2. The paper presents a novel method called MemoryGAN for GAN training. To address the two infamous problems mentioned in the paper, the authors proposed to integrate a memory network into GAN. Through memory network, MemoryGAN can explicitly learn the data distribution of real images and fake images. I think this is a very promising and meaningful extension to the original GAN. 
3. With MemoryGAN, the authors achieved best Inception Score on CIFAR-10. By ablation study, the authors demonstrated each part of the model helps to improve the final performance.
[Comments]
My comments are mainly about the experiment part:
1. In Table 2, the authors show the Inception Score of images generated by DCGAN at the last row. On CIFAR-10, it is ~5.35. As the authors mentioned, removing EM, MCGCN and Memory will result in a conventional DCGAN. However, as far as I know, DCGAN could achieve > 6.5 Inception Score in general.  I am wondering what makes such a big difference between the reported numbers in this paper and other papers?
2. In the experiments, the authors set N = 16,384, and M = 512, and z is with dimension 16. I did not understand why the memory size is such large. Take CIFAR-10 as the example, its training set contains 50k images. Using such a large memory size, each memory slot will merely count for several samples. Is a large memory size necessary to make MemoryGAN work? If not, the authors should also show ablated study on the effect of different memory size; If it is true, please explain why is that. Also, the authors should mention the training time compared with DCGAN. Updating memory with such a large size seems very time-consuming.
3. Still on the memory size in this model. I am curious about the results if the size is decreased to the same or comparable number of image categories in the training set. As the author claimed, if the memory network could learn to cluster training data into different category, we should be able to see some interesting results by sampling the keys and generate categoric images.
4. The paper should be compared with InfoGAN (Chen et al. 2016), and the authors should explain the differences between two models in the related work. Similar to MemoryGAN, InfoGAN also did not need any data annotations, but could learn the latent code flexibly.
[Summary]
This paper proposed a new model called MemoryGAN for image generation. It combined memory network with GAN, and achieved state-of-art performance on CIFAR-10. The arguments that MemoryGAN could solve the two infamous problem make sense. As I mentioned above, I did not understand why the authors used such large memory size. More explanations and experiments  should be conducted to justify this setting. Overall, I think MemoryGAN opened a new direction of GAN and worth to further explore.