Published as a conference paper at ICLR 2018 ON THE DISCRIMINATION-GENERALIZATION TRADE- OFF IN GANS
Generative adversarial training can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions, typically neural networks. The discriminator set should be large enough to be able to uniquely identify the true distribution (discriminative), and also be small enough to go beyond memorizing samples (generalizable). In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions. This is a very mild condition satisfied even by neural networks with a single neuron. Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics. When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set. When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training. Our analysis sheds lights on understanding the practical performance of GANs.
In more detail, the analysis of the paper is as follows.  Firstly, it primarily focuses on GAN objective functions which are "integral probability metrics (IPMs)"; one way to define these is by way of similarity to the W-GAN, namely IPMs replace the 1-Lipschitz functions in W-GAN with a generic set of functions F.  The paper overall avoids computational issues and treats the suprema as though exactly solved by sgd or related heuristic (the results of the paper simply state supremum, but some of the prose seems to touch on this issue).
The key arguments of the paper are as follows.
1. It argues that the discriminator set should be not simply large, it should be dense in all bounded continuous functions; as a consequence of this, the IPM is 0 iff the distributions are equal (in the weak sense).  Due to this assertion, it says that it suffices to use two layer neural networks as the discriminator set (as a consequence of the "universal approximation" results well-known in the neural network literature).
2. It argues the discriminator set should be small in order to mitigate small-sample effects.  (Together, points 1 and 2 mimic a standard bias-variance tradeoff in statistics.)  For this step, the paper relies upon standard Rademacher results plus a little bit of algebraic glue.  Curiously, the paper chooses to argue (and forms as a key tenet, indeed in the abstract) that the size of the generator set is irrelevant for this, only the size of the discriminator matters.
Unfortunately, I find significant problems with the paper, in order from most severe to least severe.
A.  The calculation ruling out the impact of the generator in generalization calculations in 2 above is flawed.  Before pointing out the concrete bug, I note that this assertion runs completely counter to intuition, and thus should be made with more explanation (as opposed to the fortunate magic it is presented as).  Moreover, I'll say that if the authors choose to "fix" this bug by adding a generator generalization term, the bound is still a remedial application of Rademacher complexity, so I'm not exactly blown away.  Anyway, the bug is as follows.  The equation which drops the role of the generator in the generalization calculation is the equation (10).  The proof of this inequality is at the start of appendix E.  Looking at the derivation in that appendix, everything is correct up to the second-to-last display, the one with a supremum over nu in G.  First of all, this right hand side should set off alarm bells; e.g., if we make the generator class big, we can make this right hand side essentially as big as the IPM allows even when mu = mu_m.  Now the bug itself appears when going to the next display: if the definition of d_F is expanded, one obtains two suprema, each own over _their own_ optimization variable (in this case the variables are discriminator functions).  When going to the next equation, the authors accidentally made the two suprema have the same variable and invoke a fortuitous but incorrect cancellation.  As stated a few sentences back, one can construct trivial counterexamples to these inequalities, for instance by making mu and mu_m arbitrarily close (even exactly equal if you wish) and then making nu arbitrarily far away and the discriminator set large enough to identify this.
B. The assertions in 1, regarding sizes of discriminator sets needed to achieve the goal of the IPM being 0 iff the distributions are equal (in the weak sense), are nothing more than immediate corollaries of approximation results well-known for decades in the neural network literature.  It is thus hard to consider this a serious contribution.
C. I will add on a non-technical note that the paper's assertion on what a good IPM "should be" is arguably misled.  There is not only a meaning to specific function classes (as with Lip_1 in Wasserstein_1) beyond simply "many functions", but moreover there is an interplay between the size of the generator set and the size of the discriminator set.  If the generator set is simple, then the discriminator set can also get away with being simple (this is dicussed in the Arora et al 2017 ICML paper, amongst other places).  Perhaps I am the one that is misled, but even so the paper does not appear to give a good justification of its standpoint.
I will conclude with typos and minor remarks.  I found the paper to contain a vast number of small errors, to the point that I doubted a single proofread.
Abstract, first line: "a minimizing"?  general grammar issue in this sentence; this sort of issue throughout the paper.
Abstract, "this is a mild condition".  Optimizing over a function class which is dense in all bounded measurable functions is not a mild assumption.  In the particular case under discussion, the size of the network can not be bounded (even though it has just two layers, or as the authors say is the span of single neurons).
Abstract, "...regardless of the size of the generator or hypothesis set".  This really needs explanation in the abstract, it is such a bold claim.  For instance, I wrote "no" in the margin while reading the abstract the first time.
Intro, first line: its -> their.
Intro, #3 "energy-based GANs": 'm' clashes with sample size.
Intro, bottom of page 1, the sentence with "irrelenvant": I can't make any sense of this sentence.
Intro, bottom of page 1, "is a much smaller discriminator set": no, the Lip_1 functions are in general incomparable to arbitrary sets of neural nets.
From here on I'll comment less on typos.
Middle of page 2, point (i): this is the only place it is argued/asserted that the discriminator set should contain essentially everything?  I think this needs a much more serious justification.
Section 1.1: Lebegure -> Lebesgue.
Page 4, vicinity of equation 5: there should really be a mention that none of these universal approximation results give a meaningful bound on the size of the network (the bound given by Barron's work, while nice, is still massive).
Start of section 3.  To be clear, while one can argue that the Lipschitz-1 constraint has a regularization effect, the reason it was originally imposed is to match the Kantorovich duality for Wasserstein_1.  Moreover I'll say this is another instance of the paper treating the discriminator set as irrelevant other than how close it is to being dense in all bounded measurable functions.