Improved Techniques for Training GANs
We present a variety of new architectural features and training procedures that we apply to the generative adversarial networks (GANs) framework. Using our new techniques, we achieve state-of-the-art results in semi-supervised classification on MNIST, CIFAR-10 and SVHN. The generated images are of high quality as confirmed by a visual Turing test: our model generates MNIST samples that humans cannot distinguish from real data, and CIFAR-10 samples that yield a human error rate of 21.3%. We also present ImageNet samples with unprecedented resolution and show that our methods enable the model to learn recognizable features of ImageNet classes.
Another of our proposed techniques, minibatch features, is based in part on ideas used for batch normalization [12], while our proposed virtual batch normalization is a direct extension of batch normalization.
One of the primary goals of this work is to improve the effectiveness of generative adversarial networks for semi-supervised learning (improving the performance of a supervised task, in this case, classification, by learning on additional unlabeled examples). Like many deep generative models, GANs have previously been applied to semi-supervised learning [13,14], and our work can be seen as a continuation and refinement of this effort. In concurrent work, Odena [15] proposes to extend GANs to predict image labels like we do in Section 5, but without our feature matching extension (Section 3.1) which we found to be critical for obtaining state-of-the-art performance.
This paper presents a collection of practical techniques for improving the training of GAN models and the quality of the samples from the learned models.  Most of the techniques are motivated heuristically, but seems to help from the experiment results.  A new application of the GAN models to semi-supervised learning is also proposed, and the proposed approach achieves great results on MNIST, CIFAR-10 and SVHN semi-supervised classification.
This paper presents a collection of techniques for improving GAN training.  Most of them are motivated heuristically, and the results are evaluated empirically.  The collection of these techniques, are effective and GAN models trained with these techniques are able to generate very good quality samples on MNIST, CIFAR-10 and SVHN.  I found the semi-supervised learning approach very interesting, and I think it should make the samples look more like examples from the classes, thus improving perceptual sample quality.  This is a natural result of optimizing the loss presented in section 5.  A big part of the improvement of sample quality may actually come from this semi-supervised learning.  However this is not verified very thoroughly.  In Table 3, the “Our methods” vs “-L” comparison seems to be verifying the use of semi-supervised learning vs not, if I understand the meaning of “L” correctly.  However, the sample quality of the “-L” model seems rather bad, and a reasonably well trained standard GAN model should do a little better, at least the obvious artifacts can be removed.  On the other hand, the experiment results show that good generative model of images do not help that much for learning classifiers in semi-supervised learning, while good classifier models learned in a semi-supervised way do not help training good generative models.  So clearly something not well understood is going on, and our intuitions might be incorrect or incomplete.  On a separate note, having achieved state-of-the-art semi-supervised learning results is nice, but from the paper it is hard to see if the methods are directly comparable.  It would be good to have some more careful control over the experiment setup, for example by making sure the classifier networks having the same architecture as previous methods, and then demonstrating the benefit of using the proposed semi-supervised training procedure.  Otherwise the picture is mixed.  The proposed Inception score metric makes intuitive sense.  However, a bad generative model that generates nothing like any of the classes, thus having p(y|x=G(z)) distributing equally across all classes, seems able to fool this metric.  It probably makes sense to measure the quality of models in similar nature using this metric, but if comparing models of different nature, for example GANs vs VAEs the results may not be very comparable as the behavior of this metric is not well understood.  The formulation of the minibatch discrimination features is a bit arbitrary for me.  I wonder if there are any reason behind actually choosing such a formulation.  Presumably there are other ways to introduce minibatch features.  At last, the paper proposes the techniques with the goal of making GAN training more stable.  But none of the experiments actually show that the GAN training is more “stable” as a result of these techniques.  The results mostly focus on quality of samples and semi-supervised learning results, but do not touch any measure of “stability”.  Maybe it would be better to change the phrasing a little bit?  Or maybe it would be more interesting to show some results demonstrating that these techniques actually improve “stability” with some understandable measure, even qualitative ones.  Overall I like the results of this paper, and some of the proposed techniques are novel, the semi-supervised learning approach is also interesting.  There are a lot of unanswered questions but some of it may be left to future work.