Approximation and Convergence Properties of Generative Adversarial Learning
Generative adversarial networks (GAN) approximate a target data distribution by jointly optimizing an objective function through a "two-player game" between a generator and a discriminator. Despite their empirical success, however, two very basic questions on how well they can approximate the target distribution remain unanswered. First, it is not known how restricting the discriminator family affects the approximation quality. Second, while a number of different objective functions have been proposed, we do not understand when convergence to the global minima of the objective function leads to convergence to the target distribution under various notions of distributional convergence. In this paper, we address these questions in a broad and unified setting by defining a notion of adversarial divergences that includes a number of recently proposed objective functions. We show that if the objective function is an adversarial divergence with some additional conditions, then using a restricted discriminator family has a moment-matching effect. Additionally, we show that for objective functions that are strict adversarial divergences, convergence in the objective function implies weak convergence, thus generalizing previous results.
The authors present a formal analysis to characterize general adversarial learning. The analysis shows that under certain conditions on the objective function the adversarial process has a moment-matching effect. They also show results on convergence properties.
The writing is quite dense and may not be accessible to most of the NIPS audience. I did not follow the full details myself. I defer to reviewers closer to the area. One general reservation I have about a formal analysis along the line presented is whether the results actually provide any insights useful in the practice of generative adversarial networks. In particular, do the results suggest properties of domains / distributions that make such domains suitable for a GAN approach, and, conversely, what practical domain properties would suggest a bad fit for GAN? Do the results provide guidelines for the choice of deep net architecture or learning parameters? Ideally, a formal analysis provides such type of feedback to the practitioners. If no such feedback can be provided, the formal results may still be of interest but unfortunately, they will remain largely a theoretical exercise. But at least, I would like to see a brief discussion on the impact on the practice of these results because it's the practical success of the GAN approach that has made it so exciting. (A similar issue arises with any formal framework that "explains" the success of deep learning. Such explanations become much more interesting if they also identify domain properties that predict where deep learning would work well and where it would not, in relation to real-world domains. I.e. what are the lessons for the practice of deep learning?)
I'm happy with the authors' response. It would be great to see the feedback incorporated in the revised paper. I've raised my rating.