Nonparametric Density Estimation with Adversarial Losses
We study minimax convergence rates of nonparametric density estimation under a large class of loss functions called "adversarial losses", which, besides classical L p losses, includes maximum mean discrepancy (MMD), Wasserstein distance, and total variation distance. These losses are closely related to the losses encoded by discriminator networks in generative adversarial networks (GANs). In a general framework, we study how the choice of loss and the assumed smoothness of the underlying density together determine the minimax rate. We also discuss implications for training GANs based on deep ReLU networks, and more general connections to learning implicit generative models in a minimax statistical sense.
under traditional statistical smoothness assumptions, we identify (i.e., prove matching upper and lower bounds on) minimax convergence rates for density estimation under several losses of the form (1). We also discuss some consequences this has for particular neural network implementations of GANs based on these losses. Finally, we study connections between minimax rates for explicit and implicit generative modeling, under a plausible notion of risk for implicit generative models.
Overview: This paper looks at nonparametric density estimation under certain classes of metrics, which the authors call "adversarial losses". To define adversarial losses, assume we have two probability distributions P and Q, and suppose X~P and Y~Q. Consider the supremum of |E[f(X)] - E[f(Y)]|, when f ranges over a class F_d of functions. This is the "adversarial loss with respect to class F_d", or simply the "F_d-metric", and generalizes several metrics, for instance the L_1 metric. Now, assume P is an unkown distribution belonging to some known class F_G, and we have n i.i.d. samples from P, and want to output an estimate another distribution Q in F_G so that the F_d-distance between P and Q is minimized. How small can we make this distance? Theorem 1 in this paper gives an upper bound for this question, and Theorem 3 gives a lower bound. Both bounds make smoothness assumptions about functions in F_d and F_G, and the bounds are tight for certain Sobolev spaces (Example 4). The upper bound is proved via an orthogonal series  estimator.  The motivation for this metric comes from generative adversarial networks (GANs), where the class F_d corresponds to functions that can be realized by a neural network with a certain architecture (the discriminator). In Theorem 7, the authors prove a certain performance guarantee for GANs (assuming perfect optimization), which improves the recent paper [25].  Finally, the paper compares the sample complexity of "explicit density estimation" as defined above, where the goal is to output a description of a distribution Q that is close to P, versus implicit density estimation, where the goal is to output a "generator function" that can generate i.i.d. samples from a distribution Q close to P. The main result here is Theorem 9, which states that, under certain assumptions, the two tasks have the same sample complexity up to constant factors. This result is the first of this spirit as far as I know.  The paper is notationally quite heavy and I haven't understood all results completely. I have also not checked any of the proofs, which appear in the supplementary material.  Pros: + given the current interest on GANs, the sample complexity result of this paper can be of significant interest.  + the notion of density estimation with respect to adversarial losses is a novel notion introduced recently, and this paper makes significant progress towards understanding this problem. In some sense this paper connects the classical density estimation literature with the new developments on generative models.  + the connection between implicit density estimation and explicit density estimation, made in this paper (Theorem 9), is novel.  + the results are mathematically interesting and quite non-trivial.  Cons: - there are lots of typos, and many of the terms have not been defined properly.   Comments for the author: * Line 26: losses -> metrics * Line 45: linear projection -> mapping * Line 49-50: delete "(1-)" * Line 66: 3 -> Section 3 * Line 80: q -> p * Line 80: please define Lebesgue norm. Also what is mu? Is mu = lambda? Please clarify. * Line 81: please define the sth-order fractional derivative, rather than referring the reader to a book. It is a central definition in your paper, which must  be standalone. * Line 85: is well-defined "for any P." * Line 85: define real-valued net. * Equation (4): hat{P} -> hat{P}_Z * Line 165: delete the second "that" * Line 172: define "parametric convergence" * Line 198: zeta^d -> (2zeta+1)^d * In Theorem 7, what is d ? * In Section 7, define the GAN setup (don't assume the reader knows what is the generator and what is the discriminator) * Line 255: on -> of * Line 265: a latent -> the latent * Line 266: missing closed brackets * Line 280: delete "below" * Line 296: no -> not * Line 318 and 319: estimators -> samplers * Section 8 is rather different from the rest of the paper. I would suggest moving it to the appendix, and using the opened up space to define all the used terms properly and elaborate more on the definitions and explain the techniques and intuition behind proofs. * In Lines 49-51 you mention many of the common metrics can be written in terms of F_d-metrics. Elaborate please. * For Theorem 1, do you need to make any assumptions about the orthogonal basis B? == After reading other reviews and authors' responses: thanks for the answers.  Definition of f^{(s)}: please mention the page number of the book on which this is defined to ease the reader. Line 26: currently the sentence reads "loss is measured not with L^p distances but rather with weaker losses" which is awkward. Lines 49--51: please mention mu is Lebesgue measure.