KDGAN: Knowledge Distillation with Generative Adversarial Networks
Knowledge distillation (KD) aims to train a lightweight classifier suitable to provide accurate inference with constrained resources in multi-label learning. Instead of directly consuming feature-label pairs, the classifier is trained by a teacher, i.e., a high-capacity model whose training may be resource-hungry. The accuracy of the classifier trained this way is usually suboptimal because it is difficult to learn the true data distribution from the teacher. An alternative method is to adversarially train the classifier against a discriminator in a two-player game akin to generative adversarial networks (GAN), which can ensure the classifier to learn the true data distribution at the equilibrium of this game. However, it may take excessively long time for such a two-player game to reach equilibrium due to high-variance gradient updates. To address these limitations, we propose a three-player game named KDGAN consisting of a classifier, a teacher, and a discriminator. The classifier and the teacher learn from each other via distillation losses and are adversarially trained against the discriminator via adversarial losses. By simultaneously optimizing the distillation and adversarial losses, the classifier will learn the true data distribution at the equilibrium. We approximate the discrete distribution learned by the classifier (or the teacher) with a concrete distribution. From the concrete distribution, we generate continuous samples to obtain low-variance gradient updates, which speed up the training. Extensive experiments using real datasets confirm the superiority of KDGAN in both accuracy and training speed.
In this paper, the authors propose combining a knowledge distillation and GANs to improve the accuracy for multi-class classification.  (In particular, the GANs are most related to IRGAN, where the discriminator is distinguishing between discrete distributions rather than continuous ones.)  At the core, they demonstrate that combining these two approaches provides a better balance of sample efficiency and convergence to the ground truth distribution for improved accuracy.  They claim two primary technical innovations (beyond combining these two approaches): using the Gumbel-Max trick for differentiability and having the classifier supervise the teacher (not just the teacher supervise the classifier).  They argue that the improvements come from lower variance gradients and that the equilibrium of the minimax game is convergence to the true label distribution.  The idea of combining these two perspectives is interesting, and both the theoretical arguments and the empirical results are compelling.  Overall, I like the paper and think others will benefit from reading it.  However, I think there are a few gaps in the analysis that if closed would greatly benefit the paper: (1) Because there are a multiple design decisions made that are claimed to balance for an improvement, it is important to show that it is not one particular change that is causing the majority of the benefit.  In particular, using the classifier to jointly train teacher has been published in "Large scale distributed neural network training through online distillation" (which should be cited but is not).  Comparing KDGAN with co-distillation would be an interesting baseline (this is missed in the hyperparameter analysis because of the way alpha parameterizes the GAN loss).  I think this is important as co-distillation even without GAN will allow for the true label distribution to be a convergence point, while maintaining low-variance updates.  (1a) While the hyperparameter sweeps provide a useful analysis of where the benefits come from, another (though in my opinion less interesting) missing component is understanding how much benefit the Gumbel-Max trick provides. (2) Lower variance of KD than GAN: Section 3.3 claims that the gradients from the KD loss are lower variance than the gradients from the GAN loss.  While this is intuitively believable, and prior work in this area is cited as support, I don't see any proof or empirical evidence adding to this claim.  There are some slight notation improvements that could be made, such as being consistent as to whether D references the discriminator or the distillation (see line 195).  However, overall the paper is clear and the results are valuable.