A Convex Duality Framework for GANs
Generative adversarial network (GAN) is a minimax game between a generator mimicking the true model and a discriminator distinguishing the samples produced by the generator from the real training samples. Given an unconstrained discriminator able to approximate any function, this game reduces to finding the generative model minimizing a divergence score, e.g. the Jensen-Shannon (JS) divergence, to the data distribution. However, in practice the discriminator is constrained to be in a smaller class F such as convolutional neural nets. Then, a natural question is how the divergence minimization interpretation will change as we constrain F. In this work, we address this question by developing a convex duality framework for analyzing GAN minimax problems. For a convex set F, this duality framework interprets the original vanilla GAN problem as finding the generative model with the minimum JS-divergence to the distributions penalized to match the moments of the data distribution, with the moments specified by the discriminators in F. We show that this interpretation more generally holds for f-GAN and Wasserstein GAN. We further apply the convex duality framework to explain why regularizing the discriminator's Lipschitz constant, e.g. via spectral normalization or gradient penalty, can greatly improve the training performance in a general f-GAN problem including the vanilla GAN formulation. We prove that Lipschitz regularization can be interpreted as convolving the original divergence score with the first-order Wasserstein distance, which results in a continuously-behaving target divergence measure. We numerically explore the power of Lipschitz regularization for improving the continuity behavior and training performance in GAN problems.
So I generally like the subject and theory of this paper very much. The authors illuminate some of the connections between many different GAN formulations and regularizations under the same convex dual framework. Unfortunately, the overall message I think will be lost on most readers, as the main bullet points of the paper seem fairly disjoint and sometimes are unclear w.r.t. GANs in the literature. For instance, when the distribution Q is introduced, it’s unclear what this is and how it is optimized. It’s finally clarified a bit in equation (21), but sadly there’s no real experiments to demonstrate that this formulation is sensible. When we finally get to equation (20), it’s unclear what this form gets us (other than eqs 21 and 22).   The biggest problem though is the lack of experimentation. I like the theory, but I’m constantly asking myself what is the consequences of all of these lemmas and whether any of this is grounded in experimentation. The authors provide some experiments on MNIST, but we’re to be led to believe that some samples might be better than others based on visualization alone (I certainly cannot tell that some sets are better than others or even good w.r.t. the target dataset). The reason we need more experimentation, particularly in this paper despite it’s contribution being mostly theoretical, is that the family of functions F is a central player in this paper and the claims being made. In neural network optimization, F is not constrained only by the architecture / regularization, but by gradient descent based optimization.  In this form, I would not recommend the paper being accepted without a better balance of theory and experimentation showing that the theory is grounded as well as to provide much needed interpretation for the reader. I would strongly encourage the authors to restructure and condense some of the theory to allow for room for this. For instance, can we explore equations 21 and 22 more with toy data? Are there ways of illuminating for the reader the importance of the 3rd player (as realized through Q)? Are there experiments that can better show the importance of the mixture model w.r.t. the duality framework?  =============== So, as I said I really like this work. The proposed changes I think sound marginally better, but what I'm hoping for is a more thorough analysis w.r.t. the method that goes beyond just inception scores (its unclear after reading the response that authors intend to do any more than this). I think for this work to have broader impact, that some additional analyses are necessary illuminating my the new objective is similar to / different from existing methods (WGAN, Roth et al., Mescheder et. al., Spectral norm, etc).  So please consider including more analysis in the final version if the paper gets accepted.